Data flow application basically means it is not computation heavy or computation is outsourced to another API. But what you do is you facilitate flow between different sources of data. It can be user data. It can be internal database. It can be external API. You collate it. You process it, and whatever the response comes, You automatically integrated into places that were in need of it.
For example, if you are making cursor, then you get data from the user chat. You get data from the code base. You get system prompts. You send it to LLM, then whatever response you get, you chunk it, Some of it goes into the code diffs. Some of it goes into the chat, and some of it ends up in context documents.
value of dataflow app = amount of hard to collate data you get * Processing and enrichment * amount of data you properly integrate automatically
what libraries you anticipate being used alot and what would be common patterns
Well, first of all, think of what types of data you will most probably encounter. It will be PDFs, it will be CSV files, it will be Excel files, it will be Word documents, it will be MD files. It can also be code files. It can be a website that is open in the browser. So we need to be able to extract data from this. Another is we will have to extract data from the chat our user is having with us, and also our context documents will be stored with us only, so we have to retrieve them also.
There are specific problems with specific file types for PDFs, It might be a complex layout from which you cannot easily extract data. It can be a website which progressively reveals data as the JavaScript executes. Or maybe the Excel file has multiple sheets in it. Now that I think of it, Excel sheet has two forms:
- The formulaic version of it
- Just the raw output
I think there might be a demand for something that does something with that.
Must-Have:
pdf-parse+pdfjs-dist(PDFs)xlsx(Excel/CSV)mammoth(Word)playwright(websites)cheerio(HTML parsing)@xenova/transformersoropenai(embeddings)pg+pgvectoror Pinecone (vector search)
Problem-Specific:
tesseract.js(OCR for scanned docs)tree-sitter(code parsing)@mozilla/readability(webpage content extraction)sharp(image processing)