Engineering
Put link data into a pipeline without breaking it
Pulling a table once is easy. Pulling it every month for a year without silent gaps is the real job.
SEOCheck research team ·

A monthly job in twelve lines
Python, standard library only. It pages through referring domains and stops on a loud error. Each response puts its rows in rows; add per (10 to 5000) for bigger pages, and the loop stops once it has read total rows.
import json, urllib.request as u
KEY='sk_live_...'
def get(**p):
q='&'.join(f'{k}={v}' for k,v in p.items())
url=f'https://seocheck.dev/api.php?{q}&key={KEY}'
with u.urlopen(url) as r:
return json.load(r)
rows, page = [], 1
while True:
d = get(action='refdomains', q='example.com', page=page)
if not d['ok']: raise SystemExit(d)
rows += d['rows']
if not d['rows'] or len(rows) >= d['total']: break
page += 1
print(len(rows), 'of', d['total'], 'balance', d['_credits']['balance'])Why seo backlink api jobs fail quietly
Not with a crash. With a table that is missing the last page. The usual cause is a loop that ends on a guess instead of on a count.
Check three things on every run. Did ok come back true? Did the row count reach total? Did _credits show the balance you expect? If any answer is no, fail the job and tell someone.
Silent success is worse than a loud failure. A job that finishes with the wrong rows looks healthy until someone reads the sheet.
Rate limits and slow responses matter less here than on many APIs. A monthly job can simply retry an hour later. What matters is that retries do not double-charge you for a call that already succeeded. Log the _credits balance before and after each run and compare. A jump larger than the job expected cost means something looped. Investigate it before the next run.
Teams that already run Airflow or a plain cron on a small server do not need anything new. A GitHub Actions schedule works too. The shape stays the same: fetch, validate, store, alert. See the cost comparison if you are weighing this against a suite export.
Common ingestion pitfalls
| Pitfall | Symptom | Fix |
|---|---|---|
| Filtering client-side after paging | Slow runs, wasted pages | Use sort and order server-side, then cut |
| Paging past the cap | Loop never ends | Deep pagination stops at 50,000 rows; plan for it |
| Treating HTTP 402 as a bug | Job retries forever | 402 means credit ran out; top up, do not retry |
| Treating 401 as transient | Same | Bad key; alert and stop |
| Comparing months blindly | Phantom drops | Index refreshes monthly; compare like with like |
| Mixing score vendors | Nonsense trends | One source per sheet |
A sensible shape for the whole thing
Keep a domain list in one place
A sheet or a table. One column of domains, nothing else at first.
Score the list with bulk_dr
Up to 2,000 domains per call, one credit. GET with domains= or POST the list as the body when it is long.
Pull details only for survivors
Run refdomains or anchors on the top slice. Two credits a call, up to 1,000 rows each.
Store snapshots, not overwrites
Date every pull. Trends need history.
Alert on change
A DR drop of several points on a vendor domain is worth a message.
Google Sheets and CORS
CORS is open, so a browser snippet or an Apps Script can call the API directly. A tiny example for a cell function. The function returns the Domain Rating of the domain in the cell.
function DR(domain){
var url='https://seocheck.dev/api.php?action=domain&q='+encodeURIComponent(domain);
var r=JSON.parse(UrlFetchApp.fetch(url).getContentText());
return r.metrics.dr;
}
Freshness and honesty
The index is rebuilt about once a month from our own crawler. Expect lag. Data older than a few weeks is normal. Do not schedule hourly pulls; they return the same thing and cost credits.
Pick the day after the refresh when you can. Otherwise run on the same day each month and accept the small lag.
Full keyword suites, daily crawls and rank tracking live elsewhere. Ahrefs and Semrush cover them; read the cost comparison before you duplicate anything. The endpoint reference has every parameter.

Designing the job so you can trust it
A pipeline is only as good as its weakest run. Build each run to leave evidence behind, so that a bad month can be explained rather than guessed at. The extra work is small. It saves a lot of argument later, and it saves credits too.
Think in four stages: fetch, validate, store and alert. Keep each one simple. Each stage should be able to fail on its own, and each failure should say which stage broke. The same four stages apply to any seo backlink api job, whatever language you write it in.
Store the raw response before you transform it
Save the JSON exactly as it came back, with the date and the call parameters in the file name or a header row. When a number looks wrong next quarter, you can go back to what the API actually said instead of to a sheet that someone edited.
Raw files cost little to keep. A spreadsheet that has been overwritten three times cannot tell you what happened in March.
Keep a run log with the row count and balance
Each run should add one line to a log: the start time, the calls made, the rows returned, the total value and the balance from _credits. The log is the first place to look when a number moves.
If the rows returned do not match total, the run is incomplete. Mark it as failed, keep the partial file, and do not merge it into the trend.
Separate retries from reruns
A retry repeats a call that failed without an answer. A rerun repeats a whole job that finished but needs to be redone. Treat them differently, because a retry of a call that actually succeeded can cost credits twice.
If you cannot tell whether a call succeeded, read the log before you send it again. That one habit prevents most surprise charges.
Alert on the failures that matter
An alert is useful only when someone acts on it. Send one for a failed key, an empty balance, a row count that does not match total and a cost far above the expected figure. Leave routine successes out of the message, or people will stop reading it.
Scheduling without wasting credits
The schedule should follow the data, not the clock. A job that runs every hour pays for the same snapshot twenty-four times a day, which is why the monthly rhythm matters so much for this kind of work.
Pick a schedule once and write it down. Changing it every month is how a trend line picks up artificial steps.
Monthly beats hourly for this data
The index is rebuilt about once a month. Pulls more often than that return the same thing and cost credits. If you need an extra run, do it after a known refresh, not in the middle of one.
One job, many domains
Put every domain in a single list and run the scoring in one pass. A bulk_dr call handles up to 2,000 domains for one credit, so a list of 1,500 prospects costs the same as a list of ten. Split the work only when a domain needs deeper data, such as refdomains.
Keep the list in one place. Two copies of a domain list drift apart, and then your trend compares different sets of sites.
Who should own the job
Give the job one owner, even if it runs on a shared server. Someone should answer when the alert fires, check the balance and decide whether a failed month needs a rerun. Without an owner, a job that fails quietly can go unnoticed for a quarter. Write the owner's name in the log header, so the next person knows whom to ask.
Test on a small list first
When you change a job, run it on five domains before you point it at the full list. A mistake in a loop shows itself in seconds on a small sample and in credits on a large one. The five-domain test costs almost nothing. Five domains is enough to show a loop.
Keep that test list in the repository next to the job, so it is the same every time you check.
When a pipeline is the wrong answer
A pipeline earns its place when you need the same numbers again and again. Pick the right tool. For a single question, a manual pull is often the better tool. Be honest about which of the two you actually need.
The other reason to skip the pipeline is the shape of the work. If the question changes every week, a fixed job will keep measuring the wrong thing.
A one-off export is not a pipeline
Say you need the referring domains of one rival for a single pitch. Run refdomains once, save the file and move on. Building a scheduled job for that would add maintenance without adding insight.
When your numbers meet other data
Link data answers one kind of question. If you need to join it with keyword rankings, conversions or call logs, keep the join in your own system and keep the link snapshot as one input. Do not expect the API to hold the other sources for you.
Decide what you keep, and for how long
Decide, before the first run, which fields you store and how long you keep them on your side. Trim the rows you do not need. A smaller file is easier to share with a client or a colleague, and easier to delete when the project ends.
The endpoint reference lists every field and the cost of each call, which makes this decision easier to write down.
Pre-flight list
- Key stored as a secret, not in the repo.
- Loop ends on row count, not on hope.
- HTTP 401 and 402 handled separately.
- Snapshots dated and kept.
- One score vendor per table.
- Monthly schedule, not hourly.