I just did the exact same thing for my company. I didn’t do the sql lite approach for gdrive though just a direct search.
The one part that is still difficult is the data modeling and table level descriptions etc. Maybe you make an update to a table - remove a column, etc. The 3rd party systems all have their schemas defined but the data warehouse is a bit more loose. So solving that really helps. Did you just use dbt schema to describe tables and columns then sync that to your bot? How did you keep it updated? And end of the day - worth building or buying? Also how did you track costs? I let users choose their model - but have learned it can get expensive fast. As I can see there are a lot of providers trying to solve this one thing. That said the data warehouse aspect is the loosely defined area and I can see dbt or one of those players try to build something.
Still can’t believe there is no true open source cms that sits on top of a Postgres db. Like. Yea you have payload but you are locked into nextjs. Just a cms that hooks into openapi or Postgres natively would be great. I still prefer Django over anything else now days.
Mostly fluff/hype. Not a value-add over just using raw MJML (which has nice VScode plugins for live previews), and in fact a long term risk to add to a codebase since react-email is just a marketing play by Resend (a startup) and will not be maintained as diligently as MJML.
This is my experience as well. MJML is the older, more reliable, better documented technology. And when it comes to debugging email rendering, you really, REALLY want as much documentation as possible.
what are the properties of the table? is the whole table 2B rows or is that the amount of row updates/inserts that happen per period (day, week, month)? We do handle a lot of large table volumes on both our products already, but there will be cost tradeoffs the larger the data volume and lower the latency you try to target.
Looks good. Question - is it always better to use a monorepo in this new AI world? Vs breaking your app into separate repos? At my company we have like 6 repos all separate nextjs apps for the same user base. Trying to consolidate to one as it should make life easier overall.
It really depends but there’s nothing stopping you from just creating a separate folder with the cloned repositories (or worktrees) that you need and having a root CLAUDE.md file that explains the directory structure and referencing the individual repo CLAUDE.md files.
A consistent and well-organised approach across the data that could be used by AI, ideally with journalling and tracking to understand how things have changed over time.
The one part that is still difficult is the data modeling and table level descriptions etc. Maybe you make an update to a table - remove a column, etc. The 3rd party systems all have their schemas defined but the data warehouse is a bit more loose. So solving that really helps. Did you just use dbt schema to describe tables and columns then sync that to your bot? How did you keep it updated? And end of the day - worth building or buying? Also how did you track costs? I let users choose their model - but have learned it can get expensive fast. As I can see there are a lot of providers trying to solve this one thing. That said the data warehouse aspect is the loosely defined area and I can see dbt or one of those players try to build something.