Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
243 views
in Technique[技术] by (71.8m points)

javascript - 如何在NodeJ中按块解析Json数据(How to parse Json data by chunks in NodeJs)

I'm creating a REST API that gets raw data from the internet then apply a REGEX to it and return it in JSON format.(我正在创建一个REST API,该API可从互联网获取原始数据,然后对其应用REGEX并以JSON格式返回。)

this is my function for getting Data as JSON.(这是我将数据作为JSON获取的功能。)
first i'm using got() to get the raw data than I apply ANIME_LIST_REGEX.exec() to filter it with the regular expression to make it in JSON format.(首先,我使用got()获取原始数据,而不是应用ANIME_LIST_REGEX.exec()以正则表达式对其进行过滤以使其成为JSON格式。) async function getAnimeList(url) { const {body} = await got(url); let ANIME_LIST_DATA = ANIME_LIST_REGEX.exec(body) if (!ANIME_LIST_DATA) { return null; } return { animeList: ANIME_LIST_DATA[1] }; }(}) in this endpoint I'm retreiving the data from the 1st function and parsing the JSON, the return it as a response.(在这个端点中,我要从第一个函数检索数据并解析JSON,然后将其作为响应返回。) app.get('/anime-list', async (req, res, next) => { const appData = await getAnimeList(URL_BASE_ANIME_LIST); var listJson = JSON5.parse(appData.animeList) res.json(listJson) }) The issue is that the returned array is pretty big (5000 entries of js objects) and the request takes long time to return and show the array(问题是返回的数组很大(js对象有5000个条目),请求需要很长时间才能返回并显示该数组)
What I want to do is to return a chunck of that array every time I call the function or reach the endpoint.(我想做的是每次我调用函数或到达端点时都返回该数组的块。)
Tried several methods but none of them made sense.(尝试了几种方法,但没有一种是有意义的。)
Does anyone got an idea?(有人知道吗?)   ask by faouzi Ch translate from so

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

You can split the large array into a small chunk with Array.prototype.splice() .(您可以使用Array.prototype.splice()将大数组拆分为小块。)

To determine the range of the chunk, you can pass queries to the endpoint.(要确定块的范围,您可以将查询传递给端点。) app.get("/anime-list", async (req, res, next) => { const appData = await getAnimeList(URL_BASE_ANIME_LIST) var listJson = JSON5.parse(appData.animeList) const from = req.query.from || 0 const count = req.query.count || 100 res.json(listJson.splice(from, count)) }) However, as others mention, calling getAnimeList() per request will cause another performance problem.(但是,正如其他人提到的那样,每个请求调用getAnimeList()会导致另一个性能问题。) I highly suggest you refactor the function to cache the result.(我强烈建议您重构该函数以缓存结果。)

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...