According to time spent for the phase “Products API – Global Data C” is exceeded more than 2 weeks requested SQL data base has been set but due to nature of the API vendors limitations lead to more time and efforts required to complete the tasks.
As the task is complex in nature and has several issues to be taken care of which appear from the api provider side, which we already have discussed here.
So we request you to check the efforts and update the job reqirements + already so much hours spent for multiple task compared to the time and efforts needed.
Task – Pool (Database):-
Initially it was assumed that this task will require less efforts and time.
The process would be to simply make cron script to fetch the data from the ‘Intelligent’ and ‘Peerless’ API.
It was assumed that the data will be received continuously and store all the data to the local db and will not have any major efforts needed.
I have sent you already reports regarding below mentioned detail about tasks, issue and bugs related tasks.
When i started the task to fetch the numbers from both inteliquent and peerless api, then i faced many issues.
Already additional efforts invested in the current task are as below but still do not have a clear indication from providers regarding data collection.
A) Fetch, analyse & debug time for api datas for both api.
B) for inteliquent api started with state plus paging wise loop but it stored only 20 to 25 thousands records stored and tested.
C) for peerless api started with state wise records and tried to get more records but it gave only 200 records per attempts & tested.
D) Initially the scripts execution put a lot of load on server, changed approach to reduced server load time.
E) conversation time regarding issues and bugs of api related and reporting of it.
Changing the process again now for collecting all the data with the new approach in order to reduce the server load time and fetching all the data from both the API, which will again require change in the logic and more efforts.
There are two sections in this pool(Database) functionalities as below;
–> Collecting Data:
=> I had set two tables namely pool_numbers(stored details of numbers) and pool_status(stored completion time of one request and status).
=> stored records state wise (5000 records per page) for intliquent and (200 records at single attempt) for peerless.
New Logic :
=> i will create new tables it will store the counts of state and keep the records count which will indicate the how many stored for single state and how many remain to store.
=> cron will run every 15 minutes to fetch data.
=> need to maintain timing gap between for both cron script of inteliquent and peerless api to reduce the server load time.
=> to implement above scenario for both api i need many efforts and time.
=> i can’t give guarantees to that all the datas will collect by this scenario because meanwhile script dosen’t response anything.
–> Updating Data:
Right now i don’t know how to get only updated data from api and then match with millions of records in local db.
i need to think login about it and help from it.
Numbers Management & Notifications Management:-
Right now I haven’t start this task. when above task is completed then i will start it soon.
If i get any issues regrading this tasks and if it will require much more efforts with respect to the budget defined, then i will let you know.