fix the unfixeble
This commit is contained in:
m
+3
-3
@@ -6,9 +6,9 @@ Read the exact versioned docs at https://docs.expo.dev/versions/v57.0.0/ before
|
|||||||
|
|
||||||
Persistence is SQLite-backed via `services/db/` (`expo-sqlite`, database `dot.db`, `user_version` = 4).
|
Persistence is SQLite-backed via `services/db/` (`expo-sqlite`, database `dot.db`, `user_version` = 4).
|
||||||
|
|
||||||
- `services/db/client.ts` — connection lifecycle: `getDatabase()`, `closeDatabase()`, `withTransaction()`.
|
- `services/db/client.ts` — connection lifecycle: `getDatabase()` (single-flight), `closeDatabase()`, `withTransaction()`. Android-only (expo-sqlite#48999, unfixed in 57.x): on a dead NPE-like connection, `recoverDatabase()` drops the poisoned handle and reopens via `openDatabaseAsync(name, { useNewConnection: true })`; `withDatabaseRetry()` wraps any `(db) → Promise<T>` with one automatic recovery+retry; `DatabaseRetryDeps` seam allows unit-testing under plain Node.
|
||||||
- `services/db/migrations.ts` — versioned, chained migrations via `PRAGMA user_version` (list of `{ version, up }`), single-flight `WeakMap` lock, `migrateDatabase(db, targetVersion?)`. v4 is a **transactional rebuild** (folders/files drop `uri` keys, gain `resource_id` + partial unique index on non-null `uri`). Migration tests: `npm run test:migrations` (better-sqlite3 harness in `tests/migrations.test.ts`); repository tests: `npm run test:db` (both suites, better-sqlite3 via the `DbSession` seam).
|
- `services/db/migrations.ts` — versioned, chained migrations via `PRAGMA user_version` (list of `{ version, up }`), single-flight `WeakMap` lock, `migrateDatabase(db, targetVersion?)`. v4 is a **transactional rebuild** (folders/files drop `uri` keys, gain `resource_id` + partial unique index on non-null `uri`). Migration tests: `npm run test:migrations` (better-sqlite3 harness in `tests/migrations.test.ts`); repository tests: `npm run test:db` (both suites, better-sqlite3 via the `DbSession` seam); recovery tests: `tests/dbClient.test.ts` (isBrokenConnectionError + withDatabaseRetry via `DatabaseRetryDeps`, included in `test:db`).
|
||||||
- `services/db/session.ts` — `DbSession` (injectable `runAsync`/`getFirstAsync`/`getAllAsync`) ; repos get it through `getSession()`, tests override it with `__setDbForTests()`. `client.ts` loads `expo-sqlite` lazily so the test suites run under plain Node.
|
- `services/db/session.ts` — `DbSession` (injectable `runAsync`/`getFirstAsync`/`getAllAsync`) ; repos get it through `getSession()`, tests override it with `__setDbForTests()`. `client.ts` loads `expo-sqlite` lazily so the test suites run under plain Node. `liveSession` wraps each method with `withDatabaseRetry` to auto-recover on the expo-sqlite NPE.
|
||||||
- `services/db/schema.ts` — `DATABASE_NAME`, `DATABASE_VERSION`, column-list constants, `DEVICE_USER_ID_KEY`, `PERMISSION_TTL_MS` (24h offline stale-cache).
|
- `services/db/schema.ts` — `DATABASE_NAME`, `DATABASE_VERSION`, column-list constants, `DEVICE_USER_ID_KEY`, `PERMISSION_TTL_MS` (24h offline stale-cache).
|
||||||
- `services/db/id.ts` — `newResourceId()` opaque 32-hex `lower(hex(randomblob(16)))`, generated per row.
|
- `services/db/id.ts` — `newResourceId()` opaque 32-hex `lower(hex(randomblob(16)))`, generated per row.
|
||||||
- `services/db/transitions.ts` — `transitionSyncStatus(from, event)`: per-row sync status transitions.
|
- `services/db/transitions.ts` — `transitionSyncStatus(from, event)`: per-row sync status transitions.
|
||||||
|
|||||||
@@ -0,0 +1,200 @@
|
|||||||
|
# Revue du codebase mobile — Retours & Actions
|
||||||
|
|
||||||
|
## Bugs confirmes (P0)
|
||||||
|
|
||||||
|
### 1. `isChildOf` trop large
|
||||||
|
- **Fichier** : `features/syncDevice.ts:39-41`
|
||||||
|
- **Probleme** : `uri.startsWith(rootUri)` matche les siblins (`Documents` matche `Documents-archive`). Le pass de reconciliation marque `exists = 0` sur le mauvais root.
|
||||||
|
- **Fix** : `uri === rootUri || uri.startsWith(rootUri + '/')`
|
||||||
|
|
||||||
|
### 2. Cache OCR casse
|
||||||
|
- **Fichier** : `hooks/useUpload.ts:35`
|
||||||
|
- **Probleme** : `setQueryData` stocke un `OcrJob` brut, mais `useOcrJob` attend `ApiData<OcrJob>` (enveloppe `{ data }`). Resultat : `TypeError` sur `.data.status`.
|
||||||
|
- **Fix** : `queryClient.setQueryData(['ocr', result.data.id], result)`
|
||||||
|
|
||||||
|
### 3. Outbox catch trop large
|
||||||
|
- **Fichier** : `features/syncOutbox.ts:46-52`
|
||||||
|
- **Probleme** : Toutes les erreurs (y compris 400 permanent) sont traitees comme transitoires. Un 400 boucle a 15s sans jamais dead-letter.
|
||||||
|
- **Fix** : Whitelist les erreurs transitoires (`NETWORK_ERROR`, `HTTP_5xx`, AbortError) ; les 4xx permanents doivent incrementer `attempts` via `markPendingOperation('failed')`.
|
||||||
|
|
||||||
|
### 4. `canAccess` nie l'acces au proprietaire avec cache expire
|
||||||
|
- **Fichier** : `services/db/repositories/permissions.ts:151-171`
|
||||||
|
- **Probleme** : Si une permission exacte est en cache mais expiree, `canAccess` retourne `allowed: false` avant d'atteindre le fallback `owner_id === deviceUserId`. Le device peut etre exclu de ses propres fichiers.
|
||||||
|
- **Fix** : Verifier le ownership (deviceUserId === owner_id) dans la lignee **avant** le short-circuit du cache expire, ou au minimum avant de retourner `allowed: false`.
|
||||||
|
|
||||||
|
### 5. `getFiles()` / `getFolderFolders()` ne filtrent pas `exists = 0`
|
||||||
|
- **Fichier** : `services/db/repositories/files.ts:75-79`, `folders.ts:96-98`
|
||||||
|
- **Probleme** : Les fichiers/dossiers supprimes du SAF restent affiches indefiniment dans l'UI.
|
||||||
|
- **Fix** : Ajouter `WHERE "exists" = 1` aux requetes, ou filtrer dans les ecrans. Attention : les rows cloud-only (`uri = NULL`) ont aussi `exists = false` — il faut distinguer "supprime du disque" de "jamais eu de copie locale". Ne pas filtrer dans le repository si le SAF walk a besoin de voir tous les rows ; filtrer dans les ecrans ou ajouter un flag `deleted_from_disk`.
|
||||||
|
|
||||||
|
### 6. Tab navigation = `router.push` sur un Stack
|
||||||
|
- **Fichier** : `components/FloatingNavBar.tsx:62`, `app/_layout.tsx`
|
||||||
|
- **Probleme** : Le layout utilise un `<Stack>` et la nav bar fait `router.push(tab.route)`. Chaque switch d'onglet empile un ecran. Le bouton systeme depile tout au lieu de quitter l'app.
|
||||||
|
- **Fix** : migrer vers le layout `<Tabs>` d'expo-router, ou utiliser `router.replace()` pour les onglets.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Bugs potentiels (P1)
|
||||||
|
|
||||||
|
### 7. Race condition dans `recoverDatabase`
|
||||||
|
- **Fichier** : `services/db/client.ts:54-68`
|
||||||
|
- **Probleme** : `database = null` avant `closeAsync` permet l'ouverture d'une connexion concurrente via `getDatabase()`.
|
||||||
|
- **Fix** : Mettre `opening` a une promise de recovery **avant** de clear `database`, ou ajouter un verrou/flag `recovering`.
|
||||||
|
|
||||||
|
### 8. Pas de single-flight sur les SAF walks
|
||||||
|
- **Fichier** : `features/syncDevice.ts`
|
||||||
|
- **Probleme** : Le loop background (`useSyncDevice`) et le bouton "Add folder" (`handlePickDirectory`) peuvent lancer `syncRoot` en parallele. Courses sur l'upsert URI unique -> `SQLITE_CONSTRAINT` -> transaction annulee.
|
||||||
|
- **Fix** : Module-level `inFlight` promise partagee par `syncRoot`/`syncDevice`.
|
||||||
|
|
||||||
|
### 9. Walk SAF dans une transaction SQLite longue
|
||||||
|
- **Fichier** : `features/syncDevice.ts:48`
|
||||||
|
- **Probleme** : `withTransaction` ouvre avant le parcours SAF (potentiellement minutes). Toute lecture DB de l'UI est bloquee.
|
||||||
|
- **Fix** : Lister d'abord tous les folders/files SAF hors transaction, puis ouvrir la transaction uniquement pour les writes DB.
|
||||||
|
|
||||||
|
### 10. `syncMode === 'manual'` se comporte comme `'full'`
|
||||||
|
- **Fichier** : `features/syncDevice.ts:127-128`
|
||||||
|
- **Probleme** : Seul `'none'` desactive le loop. Le mode `'manual'` fait aussi un walk complet toutes les 30s.
|
||||||
|
- **Fix** : Distinguer les modes ; `'manual'` ne devrait pousser que l'outbox/pas de walk SAF automatique.
|
||||||
|
|
||||||
|
### 11. Course auth 401 vs bootstrap
|
||||||
|
- **Fichier** : `context/AuthContext.tsx:32-70`
|
||||||
|
- **Probleme** : Un 401 background peut appeler `signOut()` pendant que `bootstrap` restaure le token, restaurant une session revoquee.
|
||||||
|
- **Fix** : Monotonic "session epoch" verifiee apres chaque await.
|
||||||
|
|
||||||
|
### 12. `markPendingOperation` race read-modify-write sur `attempts`
|
||||||
|
- **Fichier** : `services/db/repositories/pendingOps.ts:118-124`
|
||||||
|
- **Probleme** : Deux `SELECT attempts` concurrents lisent la meme valeur, ecrivent la meme. Le compteur ne progresse pas correctement.
|
||||||
|
- **Fix** : `UPDATE SET attempts = attempts + 1` atomique avec `RETURNING attempts`, ou `UPDATE ... WHERE id = ?` puis relire.
|
||||||
|
|
||||||
|
### 13. `refreshPermissions` race cross-account
|
||||||
|
- **Fichier** : `features/syncOutbox.ts:78-89`
|
||||||
|
- **Probleme** : Lecture de `activeUserId`, puis requete reseau, puis `saveResourcePermission` relit `getActiveUserId()` (possiblement change).
|
||||||
|
- **Fix** : Capturer `activeUserId` + token au debut du tick, les reutiliser pour toute la duree.
|
||||||
|
|
||||||
|
### 14. Migration v5 — `PRAGMA foreign_keys` hors `try/finally`
|
||||||
|
- **Fichier** : `services/db/migrations.ts:254, 259`
|
||||||
|
- **Probleme** : Si la transaction v4/v5 echoue, FK enforcement reste desactive jusqu'au restart du process.
|
||||||
|
- **Fix** : Wrapper `PRAGMA foreign_keys = OFF/ON` dans un `try/finally`.
|
||||||
|
|
||||||
|
### 15. Dead-lettered operations bloquent `pushStatus` a `'failed'` permanemment
|
||||||
|
- **Fichier** : `services/db/repositories/pendingOps.ts` + `shares.ts`/`shareLinks.ts` (`PUSH_STATUS_SQL`)
|
||||||
|
- **Probleme** : Apres `MAX_PENDING_ATTEMPTS`, l'op est `'failed'` et `PUSH_STATUS_SQL` retourne toujours `'failed'` pour la ref, meme si un opration ulterieure reussit.
|
||||||
|
- **Fix** : Ajouter un path de cleanup/cancel pour les dead-letters, ou ignorer les `'failed'` plus anciens que le plus recent `'completed'` dans le subquery.
|
||||||
|
|
||||||
|
### 16. Erreurs SAF silencieuses → faux "dossier vide"
|
||||||
|
- **Fichier** : `services/safDirectory.ts:23-25, 33-35`
|
||||||
|
- **Probleme** : `pickDirectory` et `listDirectory` catchent toutes les erreurs et retournent `null`/`[]`. Une revocation de permission SAF est interpretee comme "dossier vide" → tous les descendants marques `exists = 0` lors de la reconciliation.
|
||||||
|
- **Fix** : Distinguer "empty" (pas d'entrees) de "error" (exception lancee). Propager l'erreur ou retourner un type `{ entries: [], error?: string }`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Mauvaises pratiques
|
||||||
|
|
||||||
|
| # | Fichier | Probleme | Action |
|
||||||
|
|---|---|---|---|
|
||||||
|
| B1 | `context/AuthContext.tsx` | `signIn`/`signOut`/`continueWithoutAccount` non wrappes en `useCallback` | Ajouter `useCallback` |
|
||||||
|
| B2 | `components/FloatingNavBar.tsx:62` | `router.push(tab.route as any)` — `as any` masque les erreurs de type | Migrer vers layout `Tabs` expo-router ou typer correctement |
|
||||||
|
| B3 | `app/index.tsx:95-106` | Pas de loading/disable sur le bouton "Add folder" | Ajouter etat `syncing`, disable le bouton, spinner |
|
||||||
|
| B4 | `hooks/useUpload.ts:25-27` | `useOcrJob` poll infiniment sur `failed` | Stop sur `done` OU `failed` |
|
||||||
|
| B5 | `api/client.ts:110-126` | Timeout AbortController couvre `fetch()` mais pas `response.json()` | Passer le signal a `response.json()` ou ajouter un timeout sur la lecture body |
|
||||||
|
| B6 | 5+ fichiers | Couleurs/design tokens hardcodes | Extraire dans `constants/theme.ts` |
|
||||||
|
| B7 | `app/index.tsx` + `app/folder/[id].tsx` | `formatSize` duplique | Extraire dans `utils/format.ts` |
|
||||||
|
| B8 | `console.info`/`console.warn` | Disperses dans les chemins production | Gate derriere `__DEV__` ou logger |
|
||||||
|
| B9 | `app.json` | Nom = `"webui"` / `com.anonymous.webui` | Renommer en VaultDrop |
|
||||||
|
| B10 | `api/client.ts:18,21` | Erreurs hardcoded en francais (`'Serveur injoignable'`) | Utiliser i18n |
|
||||||
|
| B11 | `app/folder/[id].tsx:30-39` | 3 requetes DB sequentielles au focus | `Promise.all` |
|
||||||
|
| B12 | `app/login.tsx` | Password input manque `autoCapitalize="none"`, `autoCorrect={false}` | Ajouter les props |
|
||||||
|
| B13 | `app/folder/[id].tsx` | File rows Pressable mais onPress no-op | Retirer Pressable des fichiers ou ajouter une action |
|
||||||
|
| B14 | `app/_layout.tsx` + ecrans | `Stack.Screen options` defini en double (layout + ecran) | Un seul endroit |
|
||||||
|
| B15 | `features/syncDevice.ts` | Boucle sync tourne en background sans awareness `AppState` | Stopper le loop quand `AppState` est `background`, reprendre au `foreground` |
|
||||||
|
| B16 | `services/db/repositories/files.ts:20`, `folders.ts:15` | `getDeviceUserId()` appele a chaque `saveFolder`/`saveFile` (N+1, 4 round-trips par row) | Hoister `device_user_id` en parametre ou le cacher au debut du tick |
|
||||||
|
| B17 | `services/db/repositories/pendingOps.ts:100-107` | `scheduleRetries` fait N UPDATEs sequentiels | Batch `UPDATE WHERE id IN (...)` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pieces manquantes
|
||||||
|
|
||||||
|
| # | Element | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| M1 | `services/distantStorage.ts` — fichier vide | Supprimer |
|
||||||
|
| M2 | `app/search.tsx` et `app/settings.tsx` — placeholders vides | Implementer ou supprimer les onglets |
|
||||||
|
| M3 | `hooks/` — tous inutilises par les ecrans | Les brancher ou les supprimer |
|
||||||
|
| M4 | `getDeviceAuthToken`/`saveDeviceAuthToken` dans SQLite — non utilise (token dans SecureStore) | Supprimer ou marquer `@internal` |
|
||||||
|
| M5 | Aucun loading state dans les ecrans principaux | Ajouter des etats de chargement |
|
||||||
|
| M6 | Aucun error boundary React | Ajouter un `ErrorBoundary` au layout racine |
|
||||||
|
| M7 | Pas de pull-to-refresh | Ajouter `RefreshControl` |
|
||||||
|
| M8 | `AGENTS.md` mentionne v4 alors que le code est a v5 | Mettre a jour la doc |
|
||||||
|
| M9 | `localStorage.ts` — re-export legacy utilise par les ecrans | Migrer les imports vers `services/db` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Accessibilite
|
||||||
|
|
||||||
|
- **Zero proprietes d'accessibilite** dans l'app (0 matches `accessibilityRole`, `accessibilityLabel`, `aria-*`)
|
||||||
|
- Les tabs de la nav n'annoncent pas leur etat selectionne
|
||||||
|
- Les erreurs visuelles (couleur) ne sont pas announcees aux lecteurs d'ecran
|
||||||
|
- Les icons decoratives sont lues par les screen readers
|
||||||
|
|
||||||
|
**Action** : Pass d'accessibilite minimal — `accessibilityRole`, `accessibilityState`, `accessibilityLabel` sur les elements interactifs.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
| Probleme | Fichier | Impact | Action |
|
||||||
|
|----------|---------|--------|--------|
|
||||||
|
| Walk SAF dans transaction SQLite longue | `features/syncDevice.ts:48` | Bloque les writes UI pendant le walk | Lister les entries SAF hors transaction, puis transaction uniquement pour les writes |
|
||||||
|
| N+1 pattern sur `saveFolder`/`saveFile` | `services/db/repositories/files.ts:20`, `folders.ts:15` | 4 round-trips par row (`getDeviceUserId` a chaque fois) | Hoister `device_user_id` en parametre ou le cacher au debut du tick |
|
||||||
|
| `scheduleRetries` — N UPDATEs sequentiels | `services/db/repositories/pendingOps.ts:100-107` | Lent pour gros batches | Un seul `UPDATE WHERE id IN (...)` |
|
||||||
|
| 3 requetes DB sequentielles au focus | `app/folder/[id].tsx:30-39` | Lent sur gros dossiers | `Promise.all` |
|
||||||
|
| Boucle sync sans awareness AppState | `features/syncDevice.ts:125-135` | Gaspille batterie en background | Stopper le loop quand `AppState` est `background` |
|
||||||
|
| Pas de pull-to-refresh | `app/index.tsx`, `app/folder/[id].tsx` | UI ne se met a jour qu'au refocus | Ajouter `RefreshControl` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Actions planifiees (ordre de priorite)
|
||||||
|
|
||||||
|
### P0 — Corriger immediatement
|
||||||
|
- [x] 1. Fix `isChildOf` — boundary `/` (`features/syncDevice.ts`)
|
||||||
|
- [x] 2. Fix cache OCR — `{ data: result.data }` (`hooks/useUpload.ts`)
|
||||||
|
- [x] 3. Distinguer erreurs reseau / erreurs API dans l'outbox (`features/syncOutbox.ts`)
|
||||||
|
- [x] 4. Fix `canAccess` owner vs cache expire (`services/db/repositories/permissions.ts`)
|
||||||
|
- [x] 5. Filtrer `exists` dans `getFiles`/`getFolderFolders` ou les ecrans
|
||||||
|
- [ ] 6. Migrer la tab navigation vers `<Tabs>` ou `router.replace()` (`components/FloatingNavBar.tsx`)
|
||||||
|
|
||||||
|
### P1 — Fixer rapidement
|
||||||
|
- [ ] 7. Race `recoverDatabase` — serialiser l'ouverture (`services/db/client.ts`)
|
||||||
|
- [ ] 8. Single-flight sur `syncRoot`/`syncDevice` (`features/syncDevice.ts`)
|
||||||
|
- [ ] 9. `syncMode 'manual'` gate (`features/syncDevice.ts`)
|
||||||
|
- [ ] 10. `useCallback` pour `signIn`/`signOut` (`context/AuthContext.tsx`)
|
||||||
|
- [ ] 11. Atomic `attempts + 1` dans `markPendingOperation` (`services/db/repositories/pendingOps.ts`)
|
||||||
|
- [ ] 12. Course auth 401 vs bootstrap (`context/AuthContext.tsx`)
|
||||||
|
- [ ] 13. `refreshPermissions` race cross-account (`features/syncOutbox.ts`)
|
||||||
|
- [ ] 14. `PRAGMA foreign_keys` dans `try/finally` (`services/db/migrations.ts`)
|
||||||
|
- [ ] 15. Dead-letter cleanup pour `pushStatus` (`services/db/repositories/pendingOps.ts` + `shares.ts`/`shareLinks.ts`)
|
||||||
|
- [ ] 16. Erreurs SAF — distinguer "empty" de "error" (`services/safDirectory.ts`)
|
||||||
|
- [ ] 17. Awareness `AppState` sur la boucle sync (`features/syncDevice.ts`)
|
||||||
|
- [ ] 18. Hoister `device_user_id` (N+1) dans `saveFolder`/`saveFile`
|
||||||
|
- [ ] 19. Batch `scheduleRetries` (`UPDATE WHERE id IN (...)`)
|
||||||
|
|
||||||
|
### P2 — Ameliorations
|
||||||
|
- [ ] 20. Error boundary au layout racine (`app/_layout.tsx`)
|
||||||
|
- [ ] 21. Loading/disable sur "Add folder" (`app/index.tsx`)
|
||||||
|
- [ ] 22. Extraire `formatSize` en utilitaire partage (`utils/format.ts`)
|
||||||
|
- [ ] 23. Centraliser les couleurs dans un theme (`constants/theme.ts`)
|
||||||
|
- [ ] 24. `useOcrJob` — stop poll sur `failed` (`hooks/useUpload.ts`)
|
||||||
|
- [ ] 25. Password input — `autoCapitalize`, `autoCorrect` (`app/login.tsx`)
|
||||||
|
- [ ] 26. Retirer Pressable des file rows ou ajouter une action (`app/folder/[id].tsx`)
|
||||||
|
- [ ] 27. `Promise.all` pour les 3 requetes DB au focus (`app/folder/[id].tsx`)
|
||||||
|
- [ ] 28. Pass d'accessibilite minimal
|
||||||
|
- [ ] 29. Gate `console.*` derriere `__DEV__`
|
||||||
|
|
||||||
|
### P3 — Nettoyage
|
||||||
|
- [ ] 30. Supprimer `services/distantStorage.ts`
|
||||||
|
- [ ] 31. Supprimer `getDeviceAuthToken`/`saveDeviceAuthToken` de SQLite
|
||||||
|
- [ ] 32. Mettre a jour `AGENTS.md` a v5
|
||||||
|
- [ ] 33. Migrer les imports `localStorage.ts` vers `services/db`
|
||||||
|
- [ ] 34. Renommer l'app (`app.json` + `package.json`)
|
||||||
|
- [ ] 35. Localiser les messages d'erreur API
|
||||||
|
- [ ] 36. Implementer `search.tsx` et `settings.tsx` ou supprimer les onglets
|
||||||
|
- [ ] 37. Supprimer/brancher les hooks inutilises dans `hooks/`
|
||||||
@@ -0,0 +1,277 @@
|
|||||||
|
# Fix #8 + #9 : Single-flight + listing SAF hors transaction
|
||||||
|
|
||||||
|
## Diagnostic
|
||||||
|
|
||||||
|
Bug #8 (pas de single-flight) + item perf « Walk SAF dans une transaction longue »
|
||||||
|
(FEEDBACK.md:44-52). La boucle `useSyncDevice` (30s, `syncMode:'full'` par défaut)
|
||||||
|
recouvre le `syncRoot` manuel ; sur gros dossier (>30s) les deux `withTransaction`
|
||||||
|
s'entrelacent → double INSERT sur l'index partiel unique `idx_folders_uri` /
|
||||||
|
`idx_files_uri` (migrations.ts:213-214) → `SQLITE_CONSTRAINT` → transaction
|
||||||
|
annulée → walk failed silencieux (index.tsx:103).
|
||||||
|
|
||||||
|
Aucun test n'existe pour `syncRoot` / `syncDevice`.
|
||||||
|
|
||||||
|
## Architecture cible
|
||||||
|
|
||||||
|
```
|
||||||
|
syncRoot(resourceId)
|
||||||
|
├─ pre-flight : getFolder() (lecture rapide, hors garde)
|
||||||
|
└─ enqueue(() => doSyncRoot(root)) (sérialisé)
|
||||||
|
│
|
||||||
|
├─ Phase 1 : listing SAF (hors transaction)
|
||||||
|
│ listFoldersChunked()
|
||||||
|
│ for each folder → listDirectory() + yieldToMainThread()
|
||||||
|
│ → Map<uri, FileEntry[]>
|
||||||
|
│
|
||||||
|
└─ Phase 2 : writes DB (withTransaction)
|
||||||
|
for each folder → saveFolder()
|
||||||
|
for each folder → saveFile() × n
|
||||||
|
réconciliation → exists=0
|
||||||
|
→ SyncResult
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers à modifier
|
||||||
|
|
||||||
|
### 1. `services/db/client.ts` — `__setWithTransactionForTests`
|
||||||
|
|
||||||
|
`syncRoot` utilise `withTransaction` (client.ts:110) qui passe par `getDatabase()` →
|
||||||
|
expo-sqlite. Les tests Node n'ont pas expo-sqlite. Il faut pouvoir substituer la
|
||||||
|
transaction pour les tests, comme `__setDbForTests` le fait pour `getSession()`.
|
||||||
|
|
||||||
|
Ajouter une variable module-level et un setter :
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
let withTransactionOverride: (<T>(work: () => Promise<T>) => Promise<T>) | null = null;
|
||||||
|
|
||||||
|
export function __setWithTransactionForTests(
|
||||||
|
impl: (<T>(work: () => Promise<T>) => Promise<T>) | null,
|
||||||
|
): void {
|
||||||
|
withTransactionOverride = impl;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Dans `withTransaction`, si override est défini, l'utiliser sinon chemin normal.
|
||||||
|
|
||||||
|
Impact : aucun en production (`override` reste `null`). Test-only.
|
||||||
|
|
||||||
|
### 2. `features/syncDevice.ts` — refonte en deux phases + single-flight
|
||||||
|
|
||||||
|
#### 2a. Single-flight guard (module-level)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
let chain: Promise<unknown> = Promise.resolve();
|
||||||
|
|
||||||
|
function enqueue<T>(work: () => Promise<T>): Promise<T> {
|
||||||
|
const p = chain.then(work, work); // s'exécute après le précédent, même en cas d'erreur
|
||||||
|
chain = p.catch(() => {}); // la chaîne absorbe les erreurs pour ne pas bloquer le suivant
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Comportement :
|
||||||
|
- `syncRoot(A)` enqueued → démarre immédiatement (chain = resolved)
|
||||||
|
- `syncDevice()` enqueued pendant que A tourne → attend la fin de A
|
||||||
|
- Si A échoue, B démarre quand même (`.then(work, work)`)
|
||||||
|
- Les erreurs de B remontent au caller de B, pas à la chaîne
|
||||||
|
|
||||||
|
#### 2b. Extraction `doSyncRoot` (privée)
|
||||||
|
|
||||||
|
Extraire la logique actuelle de `syncRoot` dans une fonction privée
|
||||||
|
`doSyncRoot(root: StoredFolder)` qui prend un folder déjà résolu.
|
||||||
|
|
||||||
|
#### 2c. Refonte de `doSyncRoot` en deux phases
|
||||||
|
|
||||||
|
Phase 1 — Listing SAF (hors transaction, I/O disque) :
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const folders = (
|
||||||
|
await listFoldersChunked(root.uri!, { recursive: true, includeRoot: true })
|
||||||
|
).sort((a, b) => uriDepth(a.uri) - uriDepth(b.uri));
|
||||||
|
|
||||||
|
const fileEntries = new Map<string, FileEntry[]>();
|
||||||
|
let lastYield = Date.now();
|
||||||
|
for (const folder of folders) {
|
||||||
|
if (!folder.uri) continue;
|
||||||
|
if (Date.now() - lastYield >= 16) {
|
||||||
|
lastYield = Date.now();
|
||||||
|
await yieldToMainThread();
|
||||||
|
}
|
||||||
|
fileEntries.set(
|
||||||
|
folder.uri,
|
||||||
|
listDirectory(folder.uri).filter((e) => !e.isDirectory),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Phase 2 — Writes DB (withTransaction, SQL pur) :
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
return withTransaction(async () => {
|
||||||
|
const seen = new Set<string>();
|
||||||
|
const resourceIdByUri = new Map<string, string>();
|
||||||
|
let files = 0;
|
||||||
|
|
||||||
|
for (const folder of folders) {
|
||||||
|
seen.add(folder.uri);
|
||||||
|
const parentUri = dirname(folder.uri);
|
||||||
|
const parentResourceId =
|
||||||
|
folder.uri === root.uri ? null : (resourceIdByUri.get(parentUri) ?? root.resource_id);
|
||||||
|
const saved = await saveFolder(
|
||||||
|
{ uri: folder.uri, name: folder.name, exists: folder.exists },
|
||||||
|
{ parentResourceId },
|
||||||
|
);
|
||||||
|
resourceIdByUri.set(folder.uri, saved.resource_id);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const folder of folders) {
|
||||||
|
if (!folder.uri) continue;
|
||||||
|
const entries = fileEntries.get(folder.uri) ?? [];
|
||||||
|
for (const entry of entries) {
|
||||||
|
seen.add(entry.uri);
|
||||||
|
await saveFile(entry, resourceIdByUri.get(folder.uri)!);
|
||||||
|
files++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let missing = 0;
|
||||||
|
for (const folder of await getFolders()) {
|
||||||
|
if (folder.uri && isChildOf(folder.uri, root.uri!) && folder.exists && !seen.has(folder.uri)) {
|
||||||
|
await saveFolder(
|
||||||
|
{ uri: folder.uri, name: folder.name, exists: false, resource_id: folder.resource_id },
|
||||||
|
{ syncStatus: folder.syncStatus },
|
||||||
|
);
|
||||||
|
missing++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const file of await getFiles()) {
|
||||||
|
if (file.uri && isChildOf(file.uri, root.uri!) && file.exists && !seen.has(file.uri)) {
|
||||||
|
await saveFile(storedToEntry(file), file.folder_resource_id, {
|
||||||
|
resource_id: file.resource_id,
|
||||||
|
syncStatus: file.syncStatus,
|
||||||
|
});
|
||||||
|
missing++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { rootUri: root.uri!, folders: folders.length, files, missing };
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2d. Refonte des exports
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export async function syncRoot(rootResourceId: string): Promise<SyncResult> {
|
||||||
|
const root = await getFolder(rootResourceId); // pre-flight, hors garde
|
||||||
|
if (!root) throw new Error('unknown root folder');
|
||||||
|
if (!root.uri) throw new Error(`root '${root.name}' has no physical uri`);
|
||||||
|
return enqueue(() => doSyncRoot(root)); // sérialisé
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function syncDevice(): Promise<SyncResult[]> {
|
||||||
|
return enqueue(async () => { // un seul enqueue pour tout le batch
|
||||||
|
const roots = (await getFolders()).filter(
|
||||||
|
(folder) => folder.parent_resource_id === null && folder.uri !== null,
|
||||||
|
);
|
||||||
|
const results: SyncResult[] = [];
|
||||||
|
for (const root of roots) {
|
||||||
|
results.push(await doSyncRoot(root)); // pas de double enqueue
|
||||||
|
}
|
||||||
|
await checkpointDatabase();
|
||||||
|
return results;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`useSyncDevice` : inchangée. Elle appelle `syncDevice()` qui passe par `enqueue`.
|
||||||
|
|
||||||
|
### 3. `tests/syncDevice.test.ts` — nouveau fichier
|
||||||
|
|
||||||
|
Harness : combinaison des patterns existants :
|
||||||
|
- `createHarness()` + `__setDbForTests(h)` (de repositories.test.ts)
|
||||||
|
- `__setWithTransactionForTests(async (work) => work())` (nouveau, simule sans BEGIN/COMMIT)
|
||||||
|
- `__setSafWalkForTests({ list, info })` (de safWalk.test.ts)
|
||||||
|
|
||||||
|
Arbre SAF de test :
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const tree: Record<string, DirectoryEntry[]> = {
|
||||||
|
'/root': [dir('/root/a'), dir('/root/b'), file('/root/f1.txt')],
|
||||||
|
'/root/a': [dir('/root/a/x'), file('/root/a/f2.txt')],
|
||||||
|
'/root/a/x': [],
|
||||||
|
'/root/b': [file('/root/b/f3.txt')],
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests :
|
||||||
|
|
||||||
|
| # | Test | Vérifie |
|
||||||
|
|---|------|---------|
|
||||||
|
| 1 | Walk complet sur arbre moyen → comptes folders/files/missing corrects, `parent_resource_id` cohérent, `exists=1` | Fonctionnement de base |
|
||||||
|
| 2 | `syncRoot(A)` + `syncDevice()` lancés en parallèle via `Promise.all` → `maxConcurrent` (compteur dans mock `withTransaction`) = 1 | Single-flight fonctionne |
|
||||||
|
| 3 | Walk deux fois de suite → pas de doublon, pas de `SQLITE_CONSTRAINT`, mêmes `resource_id` | Idempotence |
|
||||||
|
| 4 | Walk une fois → modifier l'arbre SAF (supprimer un fichier) → walk à nouveau → `missing > 0`, le fichier a `exists=0` | Réconciliation |
|
||||||
|
| 5 | `syncDevice` avec 2 roots → les deux sont walkés, comptes corrects | Walk multi-roots |
|
||||||
|
|
||||||
|
Setup/teardown :
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
beforeEach(() => {
|
||||||
|
h = createHarness();
|
||||||
|
for (const migration of MIGRATIONS) await migration.up(h);
|
||||||
|
__setDbForTests(h);
|
||||||
|
__setWithTransactionForTests(async (work) => work());
|
||||||
|
restoreFns.push(fakeSaf());
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
__setDbForTests(null);
|
||||||
|
__setWithTransactionForTests(null);
|
||||||
|
while (restoreFns.length) restoreFns.pop()?.();
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. `package.json` — ajouter le script de test
|
||||||
|
|
||||||
|
```json
|
||||||
|
"test:syncDevice": "tsx --test tests/syncDevice.test.ts"
|
||||||
|
```
|
||||||
|
|
||||||
|
Et ajouter au script `test` :
|
||||||
|
|
||||||
|
```json
|
||||||
|
"test": "npm run test:db && npm run test:api && npm run test:sync && npm run test:saf && npm run test:syncDevice"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Ce qui ne change PAS
|
||||||
|
|
||||||
|
| Fichier | Pourquoi |
|
||||||
|
|---------|----------|
|
||||||
|
| `services/db/repositories/folders.ts` | `saveFolder`/`saveFile` inchangés — le single-flight empêche les écritures concurrentes |
|
||||||
|
| `services/db/repositories/files.ts` | Idem |
|
||||||
|
| `app/index.tsx` | `handlePickDirectory` continue d'appeler `syncRoot()` — la sérialisation est transparente |
|
||||||
|
| `features/syncDevice.types.ts` | `SyncResult` identique |
|
||||||
|
| `services/safWalk.ts` | `listFoldersChunked` + `yieldToMainThread` inchangés |
|
||||||
|
| `services/db/migrations.ts` | Aucun changement de schéma |
|
||||||
|
|
||||||
|
## Résultat attendu
|
||||||
|
|
||||||
|
| Avant | Après |
|
||||||
|
|-------|-------|
|
||||||
|
| Walk de 3min → UI gelée (reads bloqués par la transaction) | Walk de 3min → UI fluide (listing hors transaction, writes rapides) |
|
||||||
|
| `syncRoot` manuel + tick background → `SQLITE_CONSTRAINT` silencieux | Sérialisation → tick attend la fin du manuel, pas de concurrence |
|
||||||
|
|
||||||
|
## Vérification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd mobile && npx tsc --noEmit # typage
|
||||||
|
cd mobile && npm run test:db # régression repos + migrations
|
||||||
|
cd mobile && npm run test:saf # régression SAF walk
|
||||||
|
cd mobile && npm run test:syncDevice # nouveaux tests
|
||||||
|
cd mobile && npm test # tout
|
||||||
|
```
|
||||||
|
|
||||||
|
Test manuel device :
|
||||||
|
1. Ajouter un dossier vide → walk OK, files apparaissent
|
||||||
|
2. Ajouter un gros dossier (WhatsApp, >1000 fichiers) → scroll fluide pendant le walk, pas de `walk failed` dans les logs
|
||||||
|
3. Pendant le walk, attendre le tick de 30s → logs `syncDevice` et `syncRoot` jamais imbriqués (séquentiels)
|
||||||
|
4. Supprimer un fichier du dossier SAF → après prochain walk, `missing` > 0 dans les logs
|
||||||
@@ -34,8 +34,8 @@ export default function FolderScreen() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
setFolder(current);
|
setFolder(current);
|
||||||
setSubfolders(await getFolderFolders(id));
|
setSubfolders((await getFolderFolders(id)).filter(f => f.exists));
|
||||||
setFiles(await getFiles(id));
|
setFiles((await getFiles(id)).filter(f => f.exists));
|
||||||
}, [id, router]);
|
}, [id, router]);
|
||||||
|
|
||||||
useFocusEffect(
|
useFocusEffect(
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ export default function Index() {
|
|||||||
const [sections, setSections] = useState<FileSection[]>([]);
|
const [sections, setSections] = useState<FileSection[]>([]);
|
||||||
|
|
||||||
const load = useCallback(async () => {
|
const load = useCallback(async () => {
|
||||||
const files = await getFiles();
|
const files = (await getFiles()).filter(f => f.exists);
|
||||||
setSections(groupFilesByDay(files));
|
setSections(groupFilesByDay(files));
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { listDirectory, listFoldersChunked, yieldToMainThread } from '../services/safDirectory';
|
import { listEntries, listFoldersChunked, yieldToMainThread } from '../services/safWalk';
|
||||||
import type { FileEntry } from '../services/safDirectory.types';
|
import type { FileEntry } from '../services/safDirectory.types';
|
||||||
import {
|
import {
|
||||||
checkpointDatabase,
|
checkpointDatabase,
|
||||||
@@ -37,55 +37,77 @@ function storedToEntry(stored: StoredFile): FileEntry {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function isChildOf(uri: string, rootUri: string): boolean {
|
function isChildOf(uri: string, rootUri: string): boolean {
|
||||||
return uri.startsWith(rootUri);
|
return uri === rootUri || uri.startsWith(rootUri + '/');
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function syncRoot(rootResourceId: string): Promise<SyncResult> {
|
// Single-flight : toutes les marches SAF (`syncRoot` manuel et boucle de fond)
|
||||||
const root = await getFolder(rootResourceId);
|
// sont sérialisées sur une chaîne module-level. Sans cela, deux `withTransaction`
|
||||||
if (!root) throw new Error('unknown root folder');
|
// concurrents entrelacent leurs upserts et heurtent l'index partiel unique sur
|
||||||
if (!root.uri) throw new Error(`root '${root.name}' has no physical uri`);
|
// `uri` → `SQLITE_CONSTRAINT` → transaction annulée (FEEDBACK #8).
|
||||||
|
let chain: Promise<unknown> = Promise.resolve();
|
||||||
|
|
||||||
|
function enqueue<T>(work: () => Promise<T>): Promise<T> {
|
||||||
|
const p = chain.then(work, work);
|
||||||
|
chain = p.catch(() => {});
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function doSyncRoot(root: StoredFolder): Promise<SyncResult> {
|
||||||
|
const rootUri = root.uri;
|
||||||
|
if (!rootUri) throw new Error(`root '${root.name}' has no physical uri`);
|
||||||
|
|
||||||
|
// Phase 1 — listing SAF (I/O disque, HORS transaction) : la phase d'écriture
|
||||||
|
// ne doit verrouiller la base que pour les writes SQL purs, pas pendant tout
|
||||||
|
// le parcours (FEEDBACK #9 : l'UI lit la DB pendant un walk de plusieurs minutes).
|
||||||
|
const folders = (
|
||||||
|
await listFoldersChunked(rootUri, { recursive: true, includeRoot: true })
|
||||||
|
).sort((a, b) => uriDepth(a.uri) - uriDepth(b.uri));
|
||||||
|
|
||||||
|
const fileEntries = new Map<string, FileEntry[]>();
|
||||||
|
let lastYield = Date.now();
|
||||||
|
for (const folder of folders) {
|
||||||
|
if (!folder.uri) continue;
|
||||||
|
if (Date.now() - lastYield >= 16) {
|
||||||
|
lastYield = Date.now();
|
||||||
|
await yieldToMainThread();
|
||||||
|
}
|
||||||
|
fileEntries.set(
|
||||||
|
folder.uri,
|
||||||
|
listEntries(folder.uri).filter((entry) => !entry.isDirectory),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Phase 2 — writes DB (transaction courte, SQL pur).
|
||||||
return withTransaction(async () => {
|
return withTransaction(async () => {
|
||||||
const seen = new Set<string>();
|
const seen = new Set<string>();
|
||||||
|
|
||||||
const folders = (
|
|
||||||
await listFoldersChunked(root.uri as string, { recursive: true, includeRoot: true })
|
|
||||||
).sort((a, b) => uriDepth(a.uri) - uriDepth(b.uri));
|
|
||||||
|
|
||||||
const resourceIdByUri = new Map<string, string>();
|
const resourceIdByUri = new Map<string, string>();
|
||||||
const savedFolders: StoredFolder[] = [];
|
let files = 0;
|
||||||
|
|
||||||
for (const folder of folders) {
|
for (const folder of folders) {
|
||||||
seen.add(folder.uri);
|
seen.add(folder.uri);
|
||||||
const parentUri = dirname(folder.uri);
|
const parentUri = dirname(folder.uri);
|
||||||
const parentResourceId =
|
const parentResourceId =
|
||||||
folder.uri === root.uri ? null : (resourceIdByUri.get(parentUri) ?? root.resource_id);
|
folder.uri === rootUri ? null : (resourceIdByUri.get(parentUri) ?? root.resource_id);
|
||||||
const saved = await saveFolder(
|
const saved = await saveFolder(
|
||||||
{ uri: folder.uri, name: folder.name, exists: folder.exists },
|
{ uri: folder.uri, name: folder.name, exists: folder.exists },
|
||||||
{ parentResourceId },
|
{ parentResourceId },
|
||||||
);
|
);
|
||||||
resourceIdByUri.set(folder.uri, saved.resource_id);
|
resourceIdByUri.set(folder.uri, saved.resource_id);
|
||||||
savedFolders.push(saved);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
let lastYield = Date.now();
|
for (const folder of folders) {
|
||||||
let files = 0;
|
|
||||||
for (const folder of savedFolders) {
|
|
||||||
if (!folder.uri) continue;
|
if (!folder.uri) continue;
|
||||||
if (Date.now() - lastYield >= 16) {
|
const entries = fileEntries.get(folder.uri) ?? [];
|
||||||
lastYield = Date.now();
|
for (const entry of entries) {
|
||||||
await yieldToMainThread();
|
|
||||||
}
|
|
||||||
for (const entry of listDirectory(folder.uri)) {
|
|
||||||
if (entry.isDirectory) continue;
|
|
||||||
seen.add(entry.uri);
|
seen.add(entry.uri);
|
||||||
await saveFile(entry, folder.resource_id);
|
await saveFile(entry, resourceIdByUri.get(folder.uri)!);
|
||||||
files++;
|
files++;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let missing = 0;
|
let missing = 0;
|
||||||
for (const folder of await getFolders()) {
|
for (const folder of await getFolders()) {
|
||||||
if (folder.uri && isChildOf(folder.uri, root.uri as string) && folder.exists && !seen.has(folder.uri)) {
|
if (folder.uri && isChildOf(folder.uri, rootUri) && folder.exists && !seen.has(folder.uri)) {
|
||||||
await saveFolder(
|
await saveFolder(
|
||||||
{ uri: folder.uri, name: folder.name, exists: false, resource_id: folder.resource_id },
|
{ uri: folder.uri, name: folder.name, exists: false, resource_id: folder.resource_id },
|
||||||
{ syncStatus: folder.syncStatus },
|
{ syncStatus: folder.syncStatus },
|
||||||
@@ -94,7 +116,7 @@ export async function syncRoot(rootResourceId: string): Promise<SyncResult> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for (const file of await getFiles()) {
|
for (const file of await getFiles()) {
|
||||||
if (file.uri && isChildOf(file.uri, root.uri as string) && file.exists && !seen.has(file.uri)) {
|
if (file.uri && isChildOf(file.uri, rootUri) && file.exists && !seen.has(file.uri)) {
|
||||||
await saveFile(storedToEntry(file), file.folder_resource_id, {
|
await saveFile(storedToEntry(file), file.folder_resource_id, {
|
||||||
resource_id: file.resource_id,
|
resource_id: file.resource_id,
|
||||||
syncStatus: file.syncStatus,
|
syncStatus: file.syncStatus,
|
||||||
@@ -103,20 +125,28 @@ export async function syncRoot(rootResourceId: string): Promise<SyncResult> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return { rootUri: root.uri as string, folders: folders.length, files, missing };
|
return { rootUri, folders: folders.length, files, missing };
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export async function syncRoot(rootResourceId: string): Promise<SyncResult> {
|
||||||
|
const root = await getFolder(rootResourceId);
|
||||||
|
if (!root) throw new Error('unknown root folder');
|
||||||
|
return enqueue(() => doSyncRoot(root));
|
||||||
|
}
|
||||||
|
|
||||||
export async function syncDevice(): Promise<SyncResult[]> {
|
export async function syncDevice(): Promise<SyncResult[]> {
|
||||||
const roots = (await getFolders()).filter(
|
return enqueue(async () => {
|
||||||
(folder) => folder.parent_resource_id === null && folder.uri !== null,
|
const roots = (await getFolders()).filter(
|
||||||
);
|
(folder) => folder.parent_resource_id === null && folder.uri !== null,
|
||||||
const results: SyncResult[] = [];
|
);
|
||||||
for (const root of roots) {
|
const results: SyncResult[] = [];
|
||||||
results.push(await syncRoot(root.resource_id));
|
for (const root of roots) {
|
||||||
}
|
results.push(await doSyncRoot(root));
|
||||||
await checkpointDatabase();
|
}
|
||||||
return results;
|
await checkpointDatabase();
|
||||||
|
return results;
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
export function useSyncDevice(intervalMs = 30_000): () => void {
|
export function useSyncDevice(intervalMs = 30_000): () => void {
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { api, hasAuthToken } from '../api/client';
|
import { api, hasAuthToken, ApiError } from '../api/client';
|
||||||
import type { SyncOperation } from '../api/types';
|
import type { SyncOperation } from '../api/types';
|
||||||
import {
|
import {
|
||||||
getActiveUserId,
|
getActiveUserId,
|
||||||
@@ -18,6 +18,13 @@ const SYNC_BATCH_SIZE = 50;
|
|||||||
// les dead-letters prématurées.
|
// les dead-letters prématurées.
|
||||||
const TRANSIENT_RETRY_MS = 15_000;
|
const TRANSIENT_RETRY_MS = 15_000;
|
||||||
|
|
||||||
|
function isTransientError(error: unknown): boolean {
|
||||||
|
if (!(error instanceof ApiError)) return true;
|
||||||
|
if (error.code === 'NETWORK_ERROR' || error.code.startsWith('HTTP_5')) return true;
|
||||||
|
if (error.code.startsWith('HTTP_4')) return false;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
function toSyncOperation(op: PendingOperation): SyncOperation {
|
function toSyncOperation(op: PendingOperation): SyncOperation {
|
||||||
return {
|
return {
|
||||||
operation_id: op.id,
|
operation_id: op.id,
|
||||||
@@ -43,12 +50,19 @@ export async function pushPendingOps(): Promise<PushResult> {
|
|||||||
let result;
|
let result;
|
||||||
try {
|
try {
|
||||||
result = (await api.syncOps(queued.map(toSyncOperation))).data;
|
result = (await api.syncOps(queued.map(toSyncOperation))).data;
|
||||||
} catch {
|
} catch (error) {
|
||||||
// Erreur transitoire : repousser sans toucher aux attempts
|
if (isTransientError(error)) {
|
||||||
// (aucun dead-letter prématuré).
|
const retryAt = Date.now() + TRANSIENT_RETRY_MS;
|
||||||
const retryAt = Date.now() + TRANSIENT_RETRY_MS;
|
await scheduleRetries(queued.map((op) => op.id), retryAt);
|
||||||
await scheduleRetries(queued.map((op) => op.id), retryAt);
|
return { pushed: 0, retried: queued.length };
|
||||||
return { pushed: 0, retried: queued.length };
|
}
|
||||||
|
const firstOp = queued[0];
|
||||||
|
await markPendingOperation(
|
||||||
|
firstOp.id,
|
||||||
|
'failed',
|
||||||
|
error instanceof ApiError ? `${error.code}: ${error.message}` : String(error),
|
||||||
|
);
|
||||||
|
return { pushed: 0, retried: 0 };
|
||||||
}
|
}
|
||||||
|
|
||||||
// `applied` = INDEX : les opérations [0, applied) sont confirmées côté serveur.
|
// `applied` = INDEX : les opérations [0, applied) sont confirmées côté serveur.
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ export function useCreateOcrJob() {
|
|||||||
return useMutation({
|
return useMutation({
|
||||||
mutationFn: (fileId: string) => api.createOcrJob(fileId),
|
mutationFn: (fileId: string) => api.createOcrJob(fileId),
|
||||||
onSuccess: (result) => {
|
onSuccess: (result) => {
|
||||||
queryClient.setQueryData(['ocr', result.data.id], result.data);
|
queryClient.setQueryData(['ocr', result.data.id], result);
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-1
@@ -39,8 +39,9 @@
|
|||||||
"test:api": "tsx --test tests/apiClient.test.ts",
|
"test:api": "tsx --test tests/apiClient.test.ts",
|
||||||
"test:sync": "tsx --test tests/syncOutbox.test.ts",
|
"test:sync": "tsx --test tests/syncOutbox.test.ts",
|
||||||
"test:saf": "tsx --test tests/safWalk.test.ts",
|
"test:saf": "tsx --test tests/safWalk.test.ts",
|
||||||
|
"test:syncDevice": "tsx --test tests/syncDevice.test.ts",
|
||||||
"test:e2e": "tsx --test tests/e2e.live.test.ts",
|
"test:e2e": "tsx --test tests/e2e.live.test.ts",
|
||||||
"test": "npm run test:db && npm run test:api && npm run test:sync && npm run test:saf"
|
"test": "npm run test:db && npm run test:api && npm run test:sync && npm run test:saf && npm run test:syncDevice"
|
||||||
},
|
},
|
||||||
"private": true
|
"private": true
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -107,9 +107,20 @@ export async function checkpointDatabase(): Promise<void> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type WithTransactionImpl = <T>(work: (db: SQLiteDatabase) => Promise<T>) => Promise<T>;
|
||||||
|
|
||||||
|
let withTransactionOverride: WithTransactionImpl | null = null;
|
||||||
|
|
||||||
|
/** Test-only seam : les tests Node n'ont pas expo-sqlite, `syncRoot`/`syncDevice`
|
||||||
|
* substituent leur transaction par un simple `work()`. `null` en production. */
|
||||||
|
export function __setWithTransactionForTests(impl: WithTransactionImpl | null): void {
|
||||||
|
withTransactionOverride = impl;
|
||||||
|
}
|
||||||
|
|
||||||
export async function withTransaction<T>(
|
export async function withTransaction<T>(
|
||||||
work: (db: SQLiteDatabase) => Promise<T>,
|
work: (db: SQLiteDatabase) => Promise<T>,
|
||||||
): Promise<T> {
|
): Promise<T> {
|
||||||
|
if (withTransactionOverride) return withTransactionOverride(work);
|
||||||
return withDatabaseRetry(async (db) => {
|
return withDatabaseRetry(async (db) => {
|
||||||
let result!: T;
|
let result!: T;
|
||||||
await db.withTransactionAsync(async () => {
|
await db.withTransactionAsync(async () => {
|
||||||
|
|||||||
@@ -150,6 +150,9 @@ export async function canAccess(
|
|||||||
|
|
||||||
const exactCache = await getResourcePermission(resourceId, resourceType);
|
const exactCache = await getResourcePermission(resourceId, resourceType);
|
||||||
if (exactCache) {
|
if (exactCache) {
|
||||||
|
if (exactCache.ownerId === deviceUserId) {
|
||||||
|
return { allowed: true, access: 'owner', source: 'owner', stale: false, expiresAt: null };
|
||||||
|
}
|
||||||
if (exactCache.expiresAt != null && exactCache.expiresAt < now) {
|
if (exactCache.expiresAt != null && exactCache.expiresAt < now) {
|
||||||
return {
|
return {
|
||||||
allowed: false,
|
allowed: false,
|
||||||
|
|||||||
@@ -67,3 +67,9 @@ export async function listFoldersChunked(
|
|||||||
await visit(directoryUri);
|
await visit(directoryUri);
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function listEntries(directoryUri: string): DirectoryEntry[] {
|
||||||
|
const impl = walkImpl;
|
||||||
|
if (!impl) throw new Error('SafWalkImpl not configured');
|
||||||
|
return impl.list(directoryUri);
|
||||||
|
}
|
||||||
@@ -0,0 +1,276 @@
|
|||||||
|
import { test, beforeEach, afterEach } from 'node:test';
|
||||||
|
import assert from 'node:assert/strict';
|
||||||
|
import Database from 'better-sqlite3';
|
||||||
|
import { MIGRATIONS, type MigrationDb } from '../services/db/migrations';
|
||||||
|
import type { DbSession } from '../services/db/session';
|
||||||
|
import { __setDbForTests } from '../services/db/session';
|
||||||
|
import { __setWithTransactionForTests } from '../services/db/client';
|
||||||
|
import { __setSafWalkForTests } from '../services/safWalk';
|
||||||
|
import type { DirectoryEntry, Folder } from '../services/safDirectory.types';
|
||||||
|
import { syncRoot, syncDevice } from '../features/syncDevice';
|
||||||
|
import { saveFolder, getFolders, getFolderFolders, getFiles } from '../services/db';
|
||||||
|
|
||||||
|
type Harness = MigrationDb & DbSession;
|
||||||
|
|
||||||
|
let h: Harness;
|
||||||
|
let restoreFns: Array<() => void> = [];
|
||||||
|
|
||||||
|
function dir(uri: string): Folder {
|
||||||
|
return { uri, name: uri.split('/').pop() ?? uri, isDirectory: true, exists: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
function file(uri: string): DirectoryEntry {
|
||||||
|
return {
|
||||||
|
uri,
|
||||||
|
name: uri.split('/').pop() ?? uri,
|
||||||
|
isDirectory: false,
|
||||||
|
extension: '',
|
||||||
|
exists: true,
|
||||||
|
size: 1,
|
||||||
|
type: 'text/plain',
|
||||||
|
lastModified: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function info(uri: string): { uri: string; name: string; exists: boolean } {
|
||||||
|
return { uri, name: uri.split('/').pop() ?? uri, exists: true };
|
||||||
|
}
|
||||||
|
|
||||||
|
const tree: Record<string, DirectoryEntry[]> = {
|
||||||
|
'/root': [dir('/root/a'), dir('/root/b'), file('/root/f1.txt')],
|
||||||
|
'/root/a': [dir('/root/a/x'), file('/root/a/f2.txt')],
|
||||||
|
'/root/a/x': [],
|
||||||
|
'/root/b': [file('/root/b/f3.txt')],
|
||||||
|
'/rootA': [dir('/rootA/sub')],
|
||||||
|
'/rootA/sub': [],
|
||||||
|
'/rootB': [file('/rootB/fB.txt')],
|
||||||
|
};
|
||||||
|
|
||||||
|
function createHarness(): Harness {
|
||||||
|
const sqlite = new Database(':memory:');
|
||||||
|
const harness: Harness = {
|
||||||
|
execAsync: async (sql: string) => {
|
||||||
|
sqlite.exec(sql);
|
||||||
|
},
|
||||||
|
runAsync: async (sql: string, ...params: unknown[]) => {
|
||||||
|
sqlite.prepare(sql).run(...params);
|
||||||
|
},
|
||||||
|
getFirstAsync: async (sql: string, ...params: unknown[]) =>
|
||||||
|
(sqlite.prepare(sql).get(...params) ?? null) as never,
|
||||||
|
getAllAsync: async (sql: string, ...params: unknown[]) =>
|
||||||
|
sqlite.prepare(sql).all(...params) as never,
|
||||||
|
withExclusiveTransactionAsync: async (task: (txn: Harness) => Promise<void>) => {
|
||||||
|
sqlite.exec('BEGIN');
|
||||||
|
try {
|
||||||
|
await task(harness);
|
||||||
|
sqlite.exec('COMMIT');
|
||||||
|
} catch (error) {
|
||||||
|
sqlite.exec('ROLLBACK');
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return harness;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fakeSaf(): () => void {
|
||||||
|
return __setSafWalkForTests({
|
||||||
|
list: (uri) => tree[uri] ?? [],
|
||||||
|
info,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function waitUntil(condition: () => boolean, timeoutMs = 2000): Promise<void> {
|
||||||
|
const start = Date.now();
|
||||||
|
while (!condition()) {
|
||||||
|
if (Date.now() - start >= timeoutMs) throw new Error('waitUntil: timeout');
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
h = createHarness();
|
||||||
|
for (const migration of MIGRATIONS) {
|
||||||
|
await migration.up(h);
|
||||||
|
}
|
||||||
|
__setDbForTests(h);
|
||||||
|
__setWithTransactionForTests(async (work) => work({} as never));
|
||||||
|
restoreFns.push(fakeSaf());
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
__setDbForTests(null);
|
||||||
|
__setWithTransactionForTests(null);
|
||||||
|
while (restoreFns.length) restoreFns.pop()?.();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('walk complet : comptes folders/files/missing, parent_resource_id cohérent, exists=1', async () => {
|
||||||
|
const root = await saveFolder({ uri: '/root', name: 'Root' });
|
||||||
|
|
||||||
|
const result = await syncRoot(root.resource_id);
|
||||||
|
|
||||||
|
assert.deepEqual(result, { rootUri: '/root', folders: 4, files: 3, missing: 0 });
|
||||||
|
|
||||||
|
const stored = await getFiles();
|
||||||
|
assert.equal(stored.length, 3);
|
||||||
|
for (const f of stored) {
|
||||||
|
assert.equal(f.exists, true);
|
||||||
|
}
|
||||||
|
|
||||||
|
const subs = await getFolderFolders(root.resource_id);
|
||||||
|
assert.equal(subs.length, 2);
|
||||||
|
const a = subs.find((f) => f.name === 'a')!;
|
||||||
|
assert.equal(a.parent_resource_id, root.resource_id);
|
||||||
|
assert.equal(a.exists, true);
|
||||||
|
|
||||||
|
const deep = await getFolderFolders(a.resource_id);
|
||||||
|
assert.equal(deep.length, 1);
|
||||||
|
assert.equal(deep[0].name, 'x');
|
||||||
|
assert.equal(deep[0].parent_resource_id, a.resource_id);
|
||||||
|
assert.equal(deep[0].exists, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('single-flight : un second walk n’est pas lancé tant que le premier est en cours', async () => {
|
||||||
|
const rootA = await saveFolder({ uri: '/rootA', name: 'RootA' });
|
||||||
|
const rootB = await saveFolder({ uri: '/rootB', name: 'RootB' });
|
||||||
|
|
||||||
|
let release!: () => void;
|
||||||
|
const gate = new Promise<void>((resolve) => {
|
||||||
|
release = resolve;
|
||||||
|
});
|
||||||
|
let enteredYield = false;
|
||||||
|
let listingsB = 0;
|
||||||
|
|
||||||
|
restoreFns.push(
|
||||||
|
__setSafWalkForTests({
|
||||||
|
list: (uri) => {
|
||||||
|
if (uri.startsWith('/rootA') && !enteredYield) {
|
||||||
|
// Prolonge le walk de A pour dépasser le budget de 16ms et atteindre
|
||||||
|
// son premier `yield` (qui bloque sur la gate) — sinon A se termine
|
||||||
|
// trop vite et la sérialisation ne peut pas être observée.
|
||||||
|
const until = Date.now() + 20;
|
||||||
|
while (Date.now() < until) {}
|
||||||
|
}
|
||||||
|
if (uri.startsWith('/rootB')) listingsB++;
|
||||||
|
return tree[uri] ?? [];
|
||||||
|
},
|
||||||
|
yield: async () => {
|
||||||
|
enteredYield = true;
|
||||||
|
await gate;
|
||||||
|
},
|
||||||
|
info,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const pA = syncRoot(rootA.resource_id);
|
||||||
|
await waitUntil(() => enteredYield);
|
||||||
|
|
||||||
|
const pB = syncRoot(rootB.resource_id);
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||||
|
assert.equal(listingsB, 0, 'pB doit attendre la fin de pA (sérialisation des walks)');
|
||||||
|
|
||||||
|
release();
|
||||||
|
const [rA, rB] = await Promise.all([pA, pB]);
|
||||||
|
|
||||||
|
assert.equal(rA.folders, 2);
|
||||||
|
assert.equal(rB.folders, 1);
|
||||||
|
assert.ok(listingsB > 0, 'pB doit avoir walké après la libération');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('toute la phase de listing SAF précède la moindre écriture DB', async () => {
|
||||||
|
const root = await saveFolder({ uri: '/root', name: 'Root' });
|
||||||
|
|
||||||
|
let txStarted = false;
|
||||||
|
__setWithTransactionForTests(async (work) => {
|
||||||
|
txStarted = true;
|
||||||
|
return work({} as never);
|
||||||
|
});
|
||||||
|
|
||||||
|
let listCount = 0;
|
||||||
|
restoreFns.push(
|
||||||
|
__setSafWalkForTests({
|
||||||
|
list: (uri) => {
|
||||||
|
assert.equal(txStarted, false, `écriture DB pendant le listing SAF (liste ${uri})`);
|
||||||
|
listCount++;
|
||||||
|
return tree[uri] ?? [];
|
||||||
|
},
|
||||||
|
info,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const result = await syncRoot(root.resource_id);
|
||||||
|
|
||||||
|
assert.ok(listCount >= 4, `listing complet attendu, obtenu ${listCount}`);
|
||||||
|
assert.equal(result.folders, 4);
|
||||||
|
assert.equal(result.files, 3);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('walk répété : idempotent, pas de doublon ni SQLITE_CONSTRAINT', async () => {
|
||||||
|
const root = await saveFolder({ uri: '/root', name: 'Root' });
|
||||||
|
await syncRoot(root.resource_id);
|
||||||
|
|
||||||
|
const foldersBefore = new Map((await getFolders()).map((f) => [f.uri, f.resource_id]));
|
||||||
|
const filesUriBefore = new Set((await getFiles()).map((f) => f.uri));
|
||||||
|
|
||||||
|
await syncRoot(root.resource_id);
|
||||||
|
|
||||||
|
const foldersAfter = await getFolders();
|
||||||
|
assert.equal(foldersAfter.length, foldersBefore.size);
|
||||||
|
for (const f of foldersAfter) {
|
||||||
|
assert.equal(foldersBefore.get(f.uri), f.resource_id, `resource_id stable pour ${f.uri}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const filesAfter = await getFiles();
|
||||||
|
assert.equal(filesAfter.length, filesUriBefore.size);
|
||||||
|
for (const f of filesAfter) {
|
||||||
|
assert.ok(filesUriBefore.has(f.uri));
|
||||||
|
assert.equal(f.exists, true);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('réconciliation : un fichier retiré du SAF passe à exists=0 et compte en missing', async () => {
|
||||||
|
const root = await saveFolder({ uri: '/root', name: 'Root' });
|
||||||
|
await syncRoot(root.resource_id);
|
||||||
|
assert.equal((await getFiles()).length, 3);
|
||||||
|
|
||||||
|
restoreFns.push(
|
||||||
|
__setSafWalkForTests({
|
||||||
|
list: (uri) =>
|
||||||
|
uri === '/root'
|
||||||
|
? (tree['/root'] ?? []).filter((e) => e.uri !== '/root/f1.txt')
|
||||||
|
: (tree[uri] ?? []),
|
||||||
|
info,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
const result = await syncRoot(root.resource_id);
|
||||||
|
|
||||||
|
assert.equal(result.missing, 1);
|
||||||
|
const f1 = (await getFiles()).find((f) => f.uri === '/root/f1.txt')!;
|
||||||
|
assert.equal(f1.exists, false);
|
||||||
|
const remaining = (await getFiles()).filter((f) => f.exists);
|
||||||
|
assert.equal(remaining.length, 2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('syncDevice marche toutes les roots présentes', async () => {
|
||||||
|
const rootA = await saveFolder({ uri: '/rootA', name: 'RootA' });
|
||||||
|
await saveFolder({ uri: '/rootB', name: 'RootB' });
|
||||||
|
assert.equal(rootA.parent_resource_id, null);
|
||||||
|
|
||||||
|
const results = await syncDevice();
|
||||||
|
|
||||||
|
assert.equal(results.length, 2);
|
||||||
|
const byRoot = new Map(results.map((r) => [r.rootUri, r]));
|
||||||
|
assert.deepEqual(byRoot.get('/rootA'), {
|
||||||
|
rootUri: '/rootA',
|
||||||
|
folders: 2,
|
||||||
|
files: 0,
|
||||||
|
missing: 0,
|
||||||
|
});
|
||||||
|
assert.deepEqual(byRoot.get('/rootB'), {
|
||||||
|
rootUri: '/rootB',
|
||||||
|
folders: 1,
|
||||||
|
files: 1,
|
||||||
|
missing: 0,
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user