I made some in-depth tests with TFJS and I found the same problem you highlighted: when managing models with size exceeding the MB, it is crazy to send such quantity of data to the client (think about a mobile scenario with limited bandwidth).

What I cannot still understand is the reason why the project is trying to move all the logic to the frontend while could be wiser to focus and implement a single backend Rest API (using Node or Python).