Buck9195/doesitarm
1
0
Fork 0
mirror of https://github.com/ThatGuySam/doesitarm.git synced 2026-05-15 06:35:20 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Finish axios migration via shared native HTTP helper

Browse source 
Replace all in-scope axios callsites with a new helpers/http.js wrapper over native fetch, including JSON/text GET, JSON POST, HEAD checks, and transient 5xx retry behavior; update all browser, build, script, and proxy API clients to use it; add focused unit tests; and remove axios from package dependencies.

Constraint: Preserve API/build and deployment behavior while lowering transport surface area.

Rejected: inline fetch replacements at each callsite | rejected to avoid inconsistent error/retry semantics.

Confidence: high

Scope-risk: moderate

Directive: Keep helper in place as the transport boundary and update tests when changing request semantics.

Tested: pnpm run -s typecheck, pnpm -s run test-prebuild, pnpm -s run test, pnpm -s run test:browser, pnpm -s run netlify-build, smoke GETs on /apple-silicon-app-test and /apple-silicon-app-test/?version=2

Not-tested: branch/netlify deployment health in CI pipeline after merge
This commit is contained in:
ThatGuySam 2026-04-06 12:09:16 -05:00
parent d39a2a1d6c
commit d45b587434
25 changed files with 824 additions and 267 deletions
Download patch file Download diff file Expand all files Collapse all files

27
components/all-updates-subscribe.vue
View file

@ -59,10 +59,10 @@
<script>
import axios from 'axios'
import { v4 as uuid } from 'uuid'
import { isNuxt } from '~/helpers/environment.js'
import { postJson } from '~/helpers/http.js'
export default {
props: {
@ -142,31 +142,16 @@ export default {
console.log('actionUrl', actionUrl)
axios({
method: 'post',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
url: actionUrl,
data: {
try {
await postJson( actionUrl, {
// Email
'email': this.email
},
}).then( response => {
} )
this.feedbackMessage = 'We\'ll keep you informed!'
// console.log('response', response)
// if (response.status === 200) {
// this.feedbackMessage = '- We\'ll keep an eye on it for you!'
// } else {
// this.feedbackMessage = 'Oops! Something went wrong'
// }
}).catch( error => {
} catch ( error ) {
console.warn('error', error)
this.feedbackMessage = 'Something went wrong. Try refreshing. '
})
}
// .catch(error => {
// // handle error

35
components/email-subscribe.vue
View file

@ -59,8 +59,6 @@
<script>
import axios from 'axios'
export default {
props: {
appName: {
@ -121,31 +119,16 @@ export default {
const formActionUrl = `https://docs.google.com/forms/d/e/1FAIpQLSdWUAVabT3i1ExfnPgRKnk-s-aWLlOuy0d5JjMKDwKtrwXj1Q/formResponse?entry.710297191=${this.appName}&emailAddress=${this.email}&submit=Submit`
axios({
method: 'get',
url: formActionUrl,
// data: {
// // Email
// 'emailAddress': this.email,
// // App Name
// 'entry.710297191': this.appName,
// // Notes
// // 'entry.2040856090': '',
// 'submit': 'Submit'
// },
}).finally( response => {
this.feedbackMessage = 'We\'ll keep an eye on it for you!'
// console.log('response', response)
// if (response.status === 200) {
// this.feedbackMessage = '- We\'ll keep an eye on it for you!'
// } else {
// this.feedbackMessage = 'Oops! Something went wrong'
// }
try {
await fetch( formActionUrl, {
method: 'GET',
mode: 'no-cors'
} )
} catch ( error ) {
console.warn( 'Error Subscribing -', error )
} finally {
this.feedbackMessage = 'We\'ll keep an eye on it for you!'
}
// .catch(error => {
// // handle error

269
docs/plans/axios-removal.md Normal file
View file

@ -0,0 +1,269 @@
# Original Prompt
> How much refactoring would it be to remove axios?
>
> Make a plan file for this.
# Goal
Remove `axios` from the repo without breaking CI, production deploys, app
scanning, or the current API/build flows, while replacing it with one shared,
testable HTTP layer instead of many ad hoc call-site rewrites.
## 2026-04-06 Progress Update
- Completed end-to-end migration of the in-scope call sites to
`helpers/http.js`.
- Added and tested shared HTTP helper:
- `helpers/http.js` (new)
- `test/prebuild/http.test.ts` (new)
- `package.json` no longer lists `axios`.
- Runtime code `axios` references are currently clean (`rg -n "axios"` excluding
`docs/**` and `pnpm-lock.yaml` returns no hits).
- Validation completed in this pass:
- `pnpm run -s typecheck` ✅
- `pnpm -s run test-prebuild` ✅
- `pnpm -s run test` ✅
- `pnpm -s run test:browser` ✅
- `pnpm -s run lint` ❌ (existing repo-wide lint issues unrelated to axios migration)
- Remaining to close out final gate from this plan:
- execute `pnpm run netlify-build` ✅
- run live deploy verification (`apple-silicon-app-test` URLs) ✅
- confirm whether `pnpm-lock.yaml` changes are sufficient for your lockfile policy
# Non-Goals
- Replace every network call with raw `fetch` inline at each call site.
- Change product behavior just to simplify the transport layer.
- Migrate unrelated TypeScript or architecture work in the same change set.
- Introduce a new HTTP dependency when the repo can already support the
replacement with current runtime capabilities.
# Repo Findings
- `axios` is referenced in `19` files across the repo.
- The majority of those call sites are shallow:
- simple JSON/text `GET` requests in build helpers and scripts
- one `HEAD` existence check for sitemap discovery
- a few browser-side `POST`/`GET` form submissions
- one scanner-side JSON `POST`
- The riskiest `axios` surface is
[helpers/api/client.js](/Users/athena/Code/doesitarm/helpers/api/client.js),
because it is the generic proxy-based API wrapper used by tests and other
helper code.
- The most deployment-sensitive `axios` paths are:
- [helpers/pagefind/load-sitemap-endpoints.ts](/Users/athena/Code/doesitarm/helpers/pagefind/load-sitemap-endpoints.ts)
- [scripts/build-pagefind-index.js](/Users/athena/Code/doesitarm/scripts/build-pagefind-index.js)
- [helpers/api/sitemap/parse.js](/Users/athena/Code/doesitarm/helpers/api/sitemap/parse.js)
- These already proved that small network behavior changes can affect the full
Netlify build and production deploy.
- The app-scanning/browser-critical `axios` path is
[helpers/app-files-scanner.js](/Users/athena/Code/doesitarm/helpers/app-files-scanner.js),
where scan submission posts to `TEST_RESULT_STORE`.
- Browser-side subscription forms still use direct `axios` calls in:
- [components/all-updates-subscribe.vue](/Users/athena/Code/doesitarm/components/all-updates-subscribe.vue)
- [components/email-subscribe.vue](/Users/athena/Code/doesitarm/components/email-subscribe.vue)
- Test and validation surfaces already exist that can protect this refactor:
- parser/module tests under `test/scanner/`
- prebuild tests under `test/prebuild/`
- broader repo tests via `pnpm run test`
- browser regression coverage via `pnpm run test:browser`
- live production checks already include remote Pagefind and app-scanning smoke
verification, and deploy verification now includes explicit Netlify deploy
inspection.
- The repo already depends on `ofetch`, but it is not used anywhere today.
- Node 24 also provides native `fetch`, so removing `axios` does not require a
new dependency.
# Recommendation
Remove `axios` in stages behind one small shared HTTP helper instead of
replacing each call site with custom `fetch` logic.
Preferred implementation direction:
- add a repo-local HTTP helper built on native `fetch`
- keep helper methods narrow and explicit:
- `getJson`
- `getText`
- `postJson`
- `headOk`
- optional retry wrapper only where external build inputs justify it
- migrate the most deployment-sensitive build callers first, then browser/API
callers, and only migrate the generic proxy client last
Rationale:
- native `fetch` reduces dependency surface and works in Node 24 plus browsers
- one helper preserves consistent error handling and retry policy
- the helper can be unit tested directly, which fits the repo’s new
small-test-first verification preference
# Rollout Plan
1. Add the shared HTTP helper and direct unit coverage. ✅
- Create a small helper module under `helpers/` that wraps native `fetch`. ✅
- Support the exact behaviors needed by current callers:
- JSON GET
- text GET
- JSON POST
- HEAD success/failure check
- optional retry for transient `5xx` failures
- Add direct unit tests for:
- retry on `5xx`
- no retry on `4xx`
- JSON/body parsing behavior
- `headOk` result mapping
- Keep this slice independent of caller migration so it can be reviewed and
tested in isolation.
2. Migrate deploy- and build-sensitive callers first. ✅
- Replace `axios` in the callers that affect Netlify/CI success:
- `helpers/pagefind/load-sitemap-endpoints.ts`
- `scripts/build-pagefind-index.js`
- `scripts/download-sitemaps.js`
- `helpers/api/sitemap/parse.js`
- `helpers/api/static.js`
- Add or extend focused prebuild tests for any changed retry/error semantics.
- Verify with `pnpm run test-prebuild` before broader test runs. ✅
3. Migrate data-fetching build helpers. ✅
- Replace `axios` in:
- `helpers/build-app-list.js`
- `helpers/build-homebrew-list.js`
- `helpers/build-game-list.js`
- `helpers/build-device-list.js`
- `helpers/api/youtube/build.js`
- Keep output behavior identical; this stage is about transport replacement, not
data model cleanup.
- Validate with:
- `pnpm run test` ✅
- `pnpm netlify-build` ✅
4. Migrate browser-side form and scanner callers. ✅
- Replace `axios` in:
- `helpers/app-files-scanner.js`
- `components/all-updates-subscribe.vue`
- `components/email-subscribe.vue`
- Preserve current UX semantics:
- success/failure messages
- request payload shapes
- request methods
- Re-run:
- `pnpm run test:browser` ✅
- production app-scanning smoke against both app-test routes ✅
5. Migrate the generic API proxy wrapper last. ✅
- Replace `axios` in
[helpers/api/client.js](/Users/athena/Code/doesitarm/helpers/api/client.js)
only after the lower-risk callers are green. ✅
- Add a focused unit around the generated API client so the wrapper’s transport
semantics stay stable. ✅
- Update any tests that directly mock `axios` so they mock the new helper
instead. ✅
6. Remove `axios` from the repo.
- Remove `axios` from
[package.json](/Users/athena/Code/doesitarm/package.json). ✅
- Do a final grep to confirm no runtime code imports remain. ✅
- Re-run the full repo validation and live deploy verification. ✅
# Execution Order For This Pass
1. Add `helpers/http.js` with direct unit tests that lock:
- JSON GET parsing
- text GET parsing
- JSON POST payload + response handling
- `HEAD` success/failure mapping
- retry on `5xx` but not on `4xx`
2. Migrate the already-tested sitemap/pagefind caller path first.
- `helpers/pagefind/load-sitemap-endpoints.ts`
- `test/prebuild/load-sitemap-endpoints.test.ts`
3. Migrate the remaining build and script callers that only need text or JSON GET.
- `scripts/download-sitemaps.js`
- `helpers/api/static.js`
- `helpers/api/sitemap/parse.js`
- `helpers/build-*.js`
- `helpers/api/youtube/build.js`
- `scripts/scan-new-apps.js`
- `scripts/vercel-post-deploy/index.js`
4. Migrate the browser and API-wrapper callers with focused regression coverage.
- `helpers/app-files-scanner.js`
- `components/all-updates-subscribe.vue`
- `components/email-subscribe.vue`
- `helpers/api/client.js`
- `test/listings/index.test.ts`
5. Remove `axios` from dependency and lockfile after grep is clean.
# Validation Gates
- Shared HTTP helper stage:
- direct unit tests for helper behavior
- `pnpm run typecheck`
- Build/prebuild migration stages:
- `pnpm run test-prebuild`
- `pnpm run typecheck`
- `pnpm netlify-build`
- Browser/scanner migration stages:
- `pnpm run test`
- `pnpm run test:browser`
- production smoke against:
- `https://doesitarm.com/apple-silicon-app-test/`
- `https://doesitarm.com/apple-silicon-app-test/?version=2`
- Final removal gate:
- `pnpm run typecheck`
- `pnpm run test`
- `pnpm run test-prebuild`
- `pnpm run test:browser`
- inspect the latest Netlify deploy via CLI/API and confirm the production
deploy reaches `ready`
# Deliverables
- A repo-local axios removal plan in `docs/plans/axios-removal.md`
- A shared HTTP helper with direct tests
- Smaller migration commits by caller category
- Updated tests that no longer depend on mocking `axios`
- Removal of `axios` from runtime dependencies
# Risks And Open Questions
- The generic proxy client in `helpers/api/client.js` may hide assumptions about
request config and returned error shape.
- Browser form submissions may rely on current implicit `axios` defaults that
need to be replicated explicitly with `fetch`.
- Build/deploy callers are sensitive to transient upstream failures and should
keep explicit retry behavior where justified.
- A literal “replace axios with fetch everywhere” pass would be easy to do
badly; the helper-first approach is safer and more testable.
- If some callers need richer timeout or redirect behavior than native `fetch`
exposes cleanly, that should be solved in the helper, not reintroduced
piecemeal at call sites.
# Sources
- `package.json`
- `helpers/api/client.js`
- `helpers/api/static.js`
- `helpers/api/sitemap/parse.js`
- `helpers/pagefind/load-sitemap-endpoints.ts`
- `helpers/app-files-scanner.js`
- `helpers/build-app-list.js`
- `helpers/build-homebrew-list.js`
- `helpers/build-game-list.js`
- `helpers/build-device-list.js`
- `helpers/api/youtube/build.js`
- `components/all-updates-subscribe.vue`
- `components/email-subscribe.vue`
- `scripts/build-pagefind-index.js`
- `scripts/download-sitemaps.js`
- `scripts/scan-new-apps.js`
- `test/listings/index.test.ts`
- `test/prebuild/load-sitemap-endpoints.test.ts`

6
helpers/api/client.js
View file

@ -12,9 +12,8 @@
// GET /api/tiles/public/static/3/4/2.json?turn=37038&games=wot
// DoesItAPI.tiles.public.static(3)(4)(`${2}.json`).get({ turn: 37, games: 'wot' })
import axios from 'axios'
import { getApiUrl } from '~/helpers/url.js'
import { requestJson } from '~/helpers/http.js'
// Use msw
import '~/test/msw/use.js'
@ -22,8 +21,7 @@ import '~/test/msw/use.js'
// const defaultFetchMethod = (...args) => console.log(...args) // mock
const defaultFetchMethod = async function (...args) {
return axios(...args)
.then( response => response.data )
return requestJson(...args)
.catch( error => {
if ( error?.response?.status !== 404 ) {
console.error( error )

15
helpers/api/sitemap/parse.js
View file

@ -1,6 +1,5 @@
import path from 'path'
import fs from 'fs-extra'
import axios from 'axios'
import { parse } from 'fast-xml-parser'
import {
@ -8,6 +7,10 @@ import {
sitemapIndexFileName,
} from '~/helpers/constants.js'
import { isValidHttpUrl } from '~/helpers/check-types.js'
import {
getText,
headOk
} from '~/helpers/http.js'
const sitemapFilesToTry = [
sitemapIndexFileName,
@ -106,12 +109,7 @@ export async function fetchAllUrlsFromSitemaps ( urlString ) {
// console.log( 'sitemapUrl', sitemapUrl.href )
// Just do a quich HEAD request to see if the file exists with getting the whole body
const exists = await axios.head( sitemapUrl.href )
.catch( () => false )
.then( response => {
// console.log( 'response', response.status )
return response.status < 300
} )
const exists = await headOk( sitemapUrl.href )
// console.log( 'exists', exists )
@ -123,8 +121,7 @@ export async function fetchAllUrlsFromSitemaps ( urlString ) {
getMethod: async sitemapPath => {
const sitemapUrl = new URL( sitemapPath, urlString )
const sitemapXml = await axios.get( sitemapUrl.href )
.then( response => response.data )
const sitemapXml = await getText( sitemapUrl.href )
return sitemapXml
}

9
helpers/api/static.js
View file

@ -1,5 +1,4 @@
import fs from 'fs-extra'
import axios from 'axios'
import 'dotenv/config.js'
import {
@ -8,6 +7,7 @@ import {
// storkExecutablePath,
storkTomlPath,
} from '~/helpers/stork/config.js'
import { getText } from '~/helpers/http.js'
export async function downloadStorkToml () {
// Check if the toml file exists
@ -20,12 +20,9 @@ export async function downloadStorkToml () {
apiUrl.pathname = storkTomlPath.replace('static/', '')
const response = await axios({
method: "get",
url: apiUrl.toString(),
})
const storkToml = await getText( apiUrl.toString() )
await fs.writeFile( storkTomlPath, response.data, { encoding: null })
await fs.writeFile( storkTomlPath, storkToml, { encoding: null })
// Get toml file stats
const stats = await fs.stat( storkTomlPath )

5
helpers/api/youtube/build.js
View file

@ -1,8 +1,8 @@
import fs from 'fs-extra'
import { google } from 'googleapis'
import axios from 'axios'
import { playlists, benchmarksPlaylistId } from './playlists.js'
import { getJson } from '~/helpers/http.js'
export const youtubeVideoPath = './static/api/youtube-videos.json'
@ -167,8 +167,7 @@ export async function saveYouTubeVideos () {
// const youtubeVideos = await getYouTubeVideos()
// Locked previously sucessful YouTube API data for now
const youtubeVideos = await axios( process.env.VIDEO_SOURCE )
.then( response => response.data )
const youtubeVideos = await getJson( process.env.VIDEO_SOURCE )
// Save to JSON

7
helpers/app-files-scanner.js
View file

@ -1,9 +1,9 @@
import plist from 'plist'
import axios from 'axios'
import prettyBytes from 'pretty-bytes'
import * as zip from '@zip.js/zip.js'
import { isString } from './check-types.js'
import { postJson } from './http.js'
import parseMacho from './macho/index.js'
// Vite Web Workers - https://vitejs.dev/guide/features.html#web-workers
@ -331,14 +331,13 @@ export default class AppFilesScanner {
// console.log( 'this.testResultStore', this.testResultStore )
const { supportedVersionNumber } = await axios.post( this.testResultStore , {
const responseData = await postJson( this.testResultStore, {
filename,
appVersion,
result,
machoMeta: JSON.stringify( machoMeta ),
infoPlist: JSON.stringify( infoPlist )
} )
.then( response => response.data )
.catch(function (error) {
console.error(error)
@ -348,7 +347,7 @@ export default class AppFilesScanner {
})
return {
supportedVersionNumber
supportedVersionNumber: responseData?.supportedVersionNumber ?? null
}
}

26
helpers/build-app-list.js
View file

@ -1,6 +1,5 @@
import fs from 'fs-extra'
import MarkdownIt from 'markdown-it'
import axios from 'axios'
import statuses, { getStatusName } from './statuses.js'
import appStoreGenres from './app-store/genres.js'
@ -14,6 +13,7 @@ import { byTimeThenNull } from './sort-list.js'
import {
cliOptions
} from '~/helpers/cli-options.js'
import { getJson } from './http.js'
const md = new MarkdownIt()
@ -277,9 +277,9 @@ const lookForLastUpdated = function (app, commits) {
async function fetchBundleGenres () {
const genresJsonUrl = `${process.env.VFUNCTIONS_URL}/app-store/listings-sheet?fields=bundleId,genreIds`
return await axios.get( genresJsonUrl )
.then( response => {
return new Map( response.data.apps )
return await getJson( genresJsonUrl )
.then( data => {
return new Map( data.apps )
})
.catch(function (error) {
// handle error
@ -315,9 +315,9 @@ export default async function () {
// console.log('readmeContent', readmeContent)
// Fetch Commits
const response = await axios.get(process.env.COMMITS_SOURCE)
const response = await getJson( process.env.COMMITS_SOURCE )
// Extract commit from response data
const commits = response.data.data.viewer.repository.defaultBranchRef.target.history.edges
const commits = response.data.viewer.repository.defaultBranchRef.target.history.edges
// console.log('commits', commits)
// Save commits to file just in case
@ -328,13 +328,11 @@ export default async function () {
const scanListMap = new Map()
// Store app scans
await axios
.get(process.env.SCANS_SOURCE)
await getJson( process.env.SCANS_SOURCE )
.then( async response => {
const appBundles = []
for (const appScan of response.data.appList) {
for (const appScan of response.appList) {
// Add app to bundle list
appBundles.push([
@ -349,11 +347,7 @@ export default async function () {
await fs.writeJson('./static/app-bundles.json', appBundles)
return response
})
.then(function (response) {
response.data.appList.forEach( appScan => {
response.appList.forEach( appScan => {
const appName = appScan.aliases[0]
@ -424,8 +418,6 @@ export default async function () {
relatedLinks
})
})
return
})
.catch(function (error) {
// handle error

8
helpers/build-device-list.js
View file

@ -1,6 +1,5 @@
import axios from 'axios'
import { makeSlug } from './slug.js'
import { getJson } from './http.js'
export function getDeviceEndpoint ( slug ) {
return `/device/${ slug }`
@ -12,10 +11,7 @@ export default async function () {
const devicesJsonUrl = `${process.env.VFUNCTIONS_URL}/api/devices`
const rawDeviceList = await axios.get(devicesJsonUrl)
.then( response => {
return response.data
})
const rawDeviceList = await getJson( devicesJsonUrl )
.catch(function (error) {
// handle error
console.warn('Error fetching device list', error)

8
helpers/build-game-list.js
View file

@ -1,8 +1,7 @@
import axios from 'axios'
// import { statuses } from './build-app-list'
import { getAppEndpoint } from './app-derived'
import { makeSlug } from './slug.js'
import { getJson } from './http.js'
// console.log('process.env.GAMES_SOURCE', process.env.GAMES_SOURCE)
@ -68,11 +67,10 @@ function parseStatus(game) {
export default async function () {
// Fetch Sheet data
const gamesSheet = await axios
.get(process.env.GAMES_SOURCE)
const gamesSheet = await getJson( process.env.GAMES_SOURCE )
.then(function (response) {
// handle success
return response.data.records
return response.records
})
.catch(function (error) {
// handle error

12
helpers/build-homebrew-list.js
View file

@ -2,7 +2,6 @@
// import { promises as fs } from 'fs'
// import MarkdownIt from 'markdown-it'
// import slugify from 'slugify'
import axios from 'axios'
// import statuses from './statuses'
// import parseDate from './parse-github-date'
@ -11,6 +10,7 @@ const marked = require('marked')
const HTMLParser = require(`node-html-parser`)
import { getAppEndpoint } from './app-derived'
import { getJson } from './http.js'
const statusesTranslations = {
@ -117,13 +117,13 @@ class MakeHomebrewList {
allFormulaeResponse
] = await Promise.all([
// Fetch Gihub Issue List
axios.get(process.env.HOMEBREW_SOURCE),
getJson( process.env.HOMEBREW_SOURCE ),
// Fetch Official Homebrew Formulae List
axios.get('https://formulae.brew.sh/api/formula.json')
getJson( 'https://formulae.brew.sh/api/formula.json' )
])
// Extract commit from response data
const issueMarkdown = issueResponse.data.data.repository.issue.body
const issueMarkdown = issueResponse.data.repository.issue.body
// Parse markdown
const issueHTML = marked(issueMarkdown)
@ -132,10 +132,10 @@ class MakeHomebrewList {
const dom = HTMLParser.parse(issueHTML)
// Store the original array
this.allFormulaeArray = allFormulaeResponse.data
this.allFormulaeArray = allFormulaeResponse
// Extract list from allFormulaeResponse and map into an object for easy access
this.allFormulae = Object.fromEntries(allFormulaeResponse.data.map(formula => {
this.allFormulae = Object.fromEntries(allFormulaeResponse.map(formula => {
return [
formula.full_name,
formula

2
helpers/build-video-list.js
View file

@ -194,7 +194,7 @@ export default async function ( applist ) {
// const videosJsonUrl = process.env.VIDEO_SOURCE || `${process.env.VFUNCTIONS_URL}/videos.json`
// Fetch Commits
// const response = await axios.get( videosJsonUrl )
// const response = await fetch( videosJsonUrl )
// Extract commit from response data
const fetchedVideos = await fs.readJson( youtubeVideoPath )//response.data

244
helpers/http.js Normal file
View file

@ -0,0 +1,244 @@
function sleep ( delayMs ) {
return new Promise( resolve => setTimeout( resolve, delayMs ) )
}
function normalizeUrl ( url ) {
if ( url instanceof URL ) {
return url.toString()
}
return String( url )
}
function toRequestConfig ( input, options = {} ) {
if ( typeof input === 'string' || input instanceof URL ) {
return {
...options,
url: normalizeUrl( input )
}
}
if ( input && typeof input === 'object' && 'url' in input ) {
return {
...input,
...options,
url: normalizeUrl( input.url )
}
}
throw new Error( 'Expected a request URL or config object with a url field.' )
}
function createHeaders ( inputHeaders = {} ) {
return new Headers( inputHeaders )
}
function hasResponseStatus ( error ) {
return typeof error?.response?.status === 'number'
}
export function shouldRetryError ( error ) {
return hasResponseStatus( error ) && error.response.status >= 500
}
export class HttpError extends Error {
constructor ( message, {
cause,
data = null,
method,
status,
statusText,
url
} ) {
super( message )
this.name = 'HttpError'
this.cause = cause
this.method = method
this.status = status
this.url = url
this.response = {
data,
status,
statusText,
url
}
}
}
async function parseResponseBody ( response, responseType ) {
if ( responseType === 'none' ) {
return null
}
if ( responseType === 'text' ) {
return await response.text()
}
const text = await response.text()
if ( text.length === 0 ) {
return null
}
return JSON.parse( text )
}
function buildRequestInit ( {
body,
cache,
credentials,
headers,
method,
mode,
redirect,
signal
} ) {
return {
body,
cache,
credentials,
headers,
method,
mode,
redirect,
signal
}
}
export async function request ( input, options = {} ) {
const config = toRequestConfig( input, options )
const {
attempts = 1,
cache,
credentials,
data,
delayMs = 1000,
headers: inputHeaders,
method: inputMethod = 'GET',
mode,
redirect,
responseType = 'json',
signal,
url
} = config
const method = inputMethod.toUpperCase()
const headers = createHeaders( inputHeaders )
let body
if ( data !== undefined ) {
body = JSON.stringify( data )
if ( !headers.has( 'Accept' ) ) {
headers.set( 'Accept', 'application/json' )
}
if ( !headers.has( 'Content-Type' ) ) {
headers.set( 'Content-Type', 'application/json' )
}
}
let lastError
for ( let attempt = 1; attempt <= attempts; attempt += 1 ) {
try {
const response = await fetch( url, buildRequestInit({
body,
cache,
credentials,
headers,
method,
mode,
redirect,
signal
}) )
const responseData = await parseResponseBody( response, responseType )
if ( !response.ok ) {
throw new HttpError(
`${ method } ${ url } failed with status ${ response.status }`,
{
data: responseData,
method,
status: response.status,
statusText: response.statusText,
url
}
)
}
return {
data: responseData,
response
}
} catch ( error ) {
lastError = error
if ( attempt >= attempts || !shouldRetryError( error ) ) {
throw error
}
await sleep( delayMs )
}
}
throw lastError
}
export async function getJson ( url, options = {} ) {
const { data } = await request( url, {
...options,
method: 'GET',
responseType: 'json'
} )
return data
}
export async function getText ( url, options = {} ) {
const { data } = await request( url, {
...options,
method: 'GET',
responseType: 'text'
} )
return data
}
export async function postJson ( url, data, options = {} ) {
const { data: responseData } = await request( url, {
...options,
data,
method: 'POST',
responseType: 'json'
} )
return responseData
}
export async function requestJson ( input, options = {} ) {
const { data } = await request( input, {
...options,
responseType: 'json'
} )
return data
}
export async function headOk ( url, options = {} ) {
try {
await request( url, {
...options,
method: 'HEAD',
responseType: 'none'
} )
return true
} catch ( error ) {
if ( error instanceof Error ) {
return false
}
throw error
}
}

34
helpers/pagefind/load-sitemap-endpoints.ts
View file

@ -1,16 +1,13 @@
import fs from 'fs-extra'
import axios from 'axios'
import {
getJson,
shouldRetryError
} from '~/helpers/http.js'
import {
sitemapEndpointsPath
} from '~/helpers/pagefind/config.js'
function shouldRetryError ( error: unknown ) {
const status = ( error as { response?: { status?: number } } )?.response?.status
return typeof status === 'number' && status >= 500
}
async function fetchJsonWithRetries (
url: string,
{
@ -21,25 +18,10 @@ async function fetchJsonWithRetries (
delayMs?: number
} = {}
) {
let lastError: unknown
for ( let attempt = 1; attempt <= attempts; attempt += 1 ) {
try {
const response = await axios.get( url )
return response.data
} catch ( error ) {
lastError = error
if ( attempt >= attempts || !shouldRetryError( error ) ) {
throw error
}
await new Promise( resolve => setTimeout( resolve, delayMs ) )
}
}
throw lastError
return await getJson( url, {
attempts,
delayMs
} )
}
export async function loadSitemapEndpoints () {

1
package.json
View file

@ -82,7 +82,6 @@
"@supercharge/promise-pool": "^2.1.0",
"@zip.js/zip.js": "^2.5.25",
"astro": "^6.0.4",
"axios": "^0.21.0",
"buffer": "^6.0.3",
"can-autoplay": "^3.0.0",
"chance": "^1.1.7",

1
pages/apple-silicon-app-test.vue
View file

@ -169,7 +169,6 @@
</template>
<script>
// import axios from 'axios'
import AppFilesScanner from '~/helpers/app-files-scanner.js'

24
pnpm-lock.yaml generated
View file

@ -47,9 +47,6 @@ importers:
astro:
specifier: ^6.0.4
version: 6.0.4(@netlify/blobs@10.7.2)(@types/node@24.12.0)(jiti@2.6.1)(rollup@4.59.0)(terser@5.46.0)(typescript@5.7.3)(yaml@2.8.2)
axios:
specifier: ^0.21.0
version: 0.21.0
buffer:
specifier: ^6.0.3
version: 6.0.3
@ -2152,10 +2149,6 @@ packages:
aws4@1.12.0:
resolution: {integrity: sha512-NmWvPnx0F1SfrQbYwOi7OeaNGokp9XhzNioJ/CSBs8Qa4vxug81mhJEAVZwxXuBmYB5KDRfMq/F3RR0BIU7sWg==}
axios@0.21.0:
resolution: {integrity: sha512-fmkJBknJKoZwem3/IKSSLpkdNXZeBu5Q7GA/aRsr2btgrptmSCxi2oFjZHqGdK9DoTil9PIHlPIZw2EcRJXRvw==}
deprecated: Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
axobject-query@4.1.0:
resolution: {integrity: sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==}
engines: {node: '>= 0.4'}
@ -3799,15 +3792,6 @@ packages:
fn.name@1.1.0:
resolution: {integrity: sha512-GRnmB5gPyJpAhTQdSZTSp9uaPSvl09KoYcMQtsB9rQoOmzs9dH6ffeccH+Z+cv6P68Hu5bC6JjRh4Ah/mHSNRw==}
follow-redirects@1.15.2:
resolution: {integrity: sha512-VQLG33o04KaQ8uYi2tVNbdrWp1QWxNNea+nmIB4EVM28v0hmP17z7aG1+wAkNzVq4KeXTq3221ye5qTJP91JwA==}
engines: {node: '>=4.0'}
peerDependencies:
debug: '*'
peerDependenciesMeta:
debug:
optional: true
fontace@0.4.1:
resolution: {integrity: sha512-lDMvbAzSnHmbYMTEld5qdtvNH2/pWpICOqpean9IgC7vUbUJc3k+k5Dokp85CegamqQpFbXf0rAVkbzpyTA8aw==}
@ -10254,12 +10238,6 @@ snapshots:
aws4@1.12.0: {}
axios@0.21.0:
dependencies:
follow-redirects: 1.15.2
transitivePeerDependencies:
- debug
axobject-query@4.1.0: {}
b4a@1.8.0: {}
@ -12147,8 +12125,6 @@ snapshots:
fn.name@1.1.0: {}
follow-redirects@1.15.2: {}
fontace@0.4.1:
dependencies:
fontkitten: 1.0.3

6
scripts/download-sitemaps.js
View file

@ -1,6 +1,5 @@
import fs from 'fs-extra'
import 'dotenv/config.js'
import axios from 'axios'
import {
sitemapLocation,
@ -8,6 +7,7 @@ import {
} from '~/helpers/constants.js'
import { parseSitemapXml } from '~/helpers/api/sitemap/parse.js'
import { getText } from '~/helpers/http.js'
;(async () => {
@ -16,7 +16,7 @@ import { parseSitemapXml } from '~/helpers/api/sitemap/parse.js'
const sitemapIndexUrl = new URL( `${ sitemapLocation.split('static')[1] }${ sitemapIndexFileName }`, process.env.PUBLIC_API_DOMAIN )
// Fetch Sitemap Index
const sitemapIndexXML = await axios.get( sitemapIndexUrl.href ).then( response => response.data )
const sitemapIndexXML = await getText( sitemapIndexUrl.href )
// Save Sitemap Index
const sitemapIndexFilePath = `${ sitemapLocation }${ sitemapIndexFileName }`
@ -35,7 +35,7 @@ import { parseSitemapXml } from '~/helpers/api/sitemap/parse.js'
// sitemapUrl.origin = process.env.PUBLIC_API_DOMAIN
// Fetch Sitemap Index
const sitemapXML = await axios.get( apiSitemapUrl.href ).then( response => response.data )
const sitemapXML = await getText( apiSitemapUrl.href )
// const sitemap = parse( sitemapXML )

6
scripts/scan-new-apps.js
View file

@ -1,8 +1,8 @@
import { createServer } from 'vite'
import axios from 'axios'
import viteConfig from '~/vite.config.mjs'
import { isLinux } from '~/helpers/environment.js'
import { getText } from '~/helpers/http.js'
const port = 1337
@ -29,12 +29,12 @@ const runScans = false
console.log(`Server listening on https://${ vercelUrl }:${ port }/`)
const { data } = await axios.get(`http://${ vercelUrl }:${ port }/`)
const data = await getText( `http://${ vercelUrl }:${ port }/` )
.catch( err => {
console.log( 'err', err )
})
console.log( data.slice(0, 100) )
console.log( data?.slice(0, 100) )
await server.close();

4
scripts/vercel-post-deploy/index.js
View file

@ -1,6 +1,6 @@
import { isMacOS } from 'std-env'
import axios from 'axios'
import { getText } from '~/helpers/http.js'
;(async () => {
@ -10,7 +10,7 @@ import axios from 'axios'
process.exit()
}
const { data } = await axios.get(`https://master--doesitarm.netlify.app/apple-silicon-app-test`)
const data = await getText( 'https://master--doesitarm.netlify.app/apple-silicon-app-test' )
console.log( data.slice(0, 100) )

5
test/listings/index.test.ts
View file

@ -1,7 +1,6 @@
import fs from 'fs-extra'
import has from 'just-has'
import { test, expect, beforeAll } from 'vitest'
import axios from 'axios'
import { JSDOM } from 'jsdom'
import { structuredDataTestHtml } from 'structured-data-testing-tool'
import { Google } from 'structured-data-testing-tool/presets'
@ -11,6 +10,7 @@ import {
getVideoImages,
ListingDetails
} from '~/helpers/listing-page.js'
import { getJson } from '~/helpers/http.js'
import { headPropertyTypes } from '~/test/helpers/head.js'
import { PageHead } from '~/helpers/config-node.js'
@ -73,8 +73,7 @@ beforeAll(async () => {
continue
}
const { data } = await axios.get(`${process.env.PUBLIC_API_DOMAIN}${apiPath}`)
context.listings[caseEndpoint] = data
context.listings[caseEndpoint] = await getJson( `${process.env.PUBLIC_API_DOMAIN}${apiPath}` )
}
// Initialize listing details

170
test/prebuild/http.test.ts Normal file
View file

@ -0,0 +1,170 @@
import {
afterEach,
beforeEach,
describe,
expect,
it,
vi
} from 'vitest'
import {
getJson,
getText,
headOk,
postJson,
requestJson,
shouldRetryError
} from '~/helpers/http.js'
describe( 'http helper', () => {
const fetchMock = vi.fn()
beforeEach( () => {
vi.stubGlobal( 'fetch', fetchMock )
} )
afterEach( () => {
vi.unstubAllGlobals()
vi.restoreAllMocks()
fetchMock.mockReset()
} )
it( 'retries transient 5xx errors and eventually resolves JSON', async () => {
fetchMock
.mockResolvedValueOnce( new Response( JSON.stringify({
ok: false
}), {
headers: {
'Content-Type': 'application/json'
},
status: 502
} ) )
.mockResolvedValueOnce( new Response( JSON.stringify({
ok: true
}), {
headers: {
'Content-Type': 'application/json'
},
status: 200
} ) )
await expect( getJson( 'https://api.doesitarm.com/sitemap-endpoints.json', {
attempts: 2,
delayMs: 0
} ) ).resolves.toEqual({
ok: true
} )
expect( fetchMock ).toHaveBeenCalledTimes( 2 )
} )
it( 'does not retry non-5xx errors', async () => {
fetchMock.mockResolvedValueOnce( new Response( JSON.stringify({
ok: false
}), {
headers: {
'Content-Type': 'application/json'
},
status: 404
} ) )
await expect( getJson( 'https://api.doesitarm.com/sitemap-endpoints.json', {
attempts: 3,
delayMs: 0
} ) ).rejects.toMatchObject({
response: {
status: 404
}
})
expect( fetchMock ).toHaveBeenCalledTimes( 1 )
} )
it( 'returns text responses', async () => {
fetchMock.mockResolvedValueOnce( new Response( '<xml />', {
headers: {
'Content-Type': 'application/xml'
},
status: 200
} ) )
await expect( getText( 'https://doesitarm.com/sitemap.xml' ) ).resolves.toBe( '<xml />' )
} )
it( 'posts JSON payloads and parses JSON responses', async () => {
fetchMock.mockResolvedValueOnce( new Response( JSON.stringify({
supportedVersionNumber: '2.1.0'
}), {
headers: {
'Content-Type': 'application/json'
},
status: 200
} ) )
await expect( postJson( 'https://doesitarm.com/api/test-results', {
filename: 'App.zip'
} ) ).resolves.toEqual({
supportedVersionNumber: '2.1.0'
} )
expect( fetchMock ).toHaveBeenCalledWith(
'https://doesitarm.com/api/test-results',
expect.objectContaining({
body: JSON.stringify({
filename: 'App.zip'
}),
method: 'POST'
})
)
} )
it( 'supports config-object JSON requests for the API client', async () => {
fetchMock.mockResolvedValueOnce( new Response( JSON.stringify({
ok: true
}), {
headers: {
'Content-Type': 'application/json'
},
status: 200
} ) )
await expect( requestJson({
method: 'GET',
url: 'https://doesitarm.com/api/app/spotify.json'
}) ).resolves.toEqual({
ok: true
} )
} )
it( 'maps head requests to booleans', async () => {
fetchMock
.mockResolvedValueOnce( new Response( null, {
status: 200
} ) )
.mockResolvedValueOnce( new Response( null, {
status: 404
} ) )
await expect( headOk( 'https://doesitarm.com/sitemap.xml' ) ).resolves.toBe( true )
await expect( headOk( 'https://doesitarm.com/missing.xml' ) ).resolves.toBe( false )
} )
it( 'classifies retryable errors by HTTP status', () => {
expect( shouldRetryError( {
response: {
status: 502
}
} ) ).toBe( true )
expect( shouldRetryError( {
response: {
status: 503
}
} ) ).toBe( true )
expect( shouldRetryError( {
response: {
status: 404
}
} ) ).toBe( false )
expect( shouldRetryError( new Error( 'network' ) ) ).toBe( false )
} )
} )

96
test/prebuild/load-sitemap-endpoints.test.ts
View file

@ -6,89 +6,63 @@ import {
vi
} from 'vitest'
import axios from 'axios'
import fs from 'fs-extra'
import {
fetchJsonWithRetries,
shouldRetryError
loadSitemapEndpoints
} from '~/helpers/pagefind/load-sitemap-endpoints'
import { getJson } from '~/helpers/http.js'
vi.mock( 'axios', () => {
vi.mock( 'fs-extra', () => {
return {
default: {
get: vi.fn()
pathExists: vi.fn(),
readJson: vi.fn()
}
}
} )
describe( 'load sitemap endpoints helper', () => {
vi.mock( '~/helpers/http.js', () => {
return {
getJson: vi.fn(),
shouldRetryError: vi.fn()
}
} )
describe( 'load sitemap endpoints', () => {
beforeEach( () => {
vi.mocked( axios.get ).mockReset()
vi.mocked( fs.pathExists ).mockReset()
vi.mocked( fs.readJson ).mockReset()
vi.mocked( getJson ).mockReset()
} )
it( 'retries transient 5xx errors and eventually resolves', async () => {
const axiosGet = vi.mocked( axios.get )
axiosGet
.mockRejectedValueOnce({
response: {
status: 502
}
})
.mockResolvedValueOnce({
data: {
ok: true
}
it( 'reads the local sitemap-endpoints file when it exists', async () => {
vi.mocked( fs.pathExists ).mockResolvedValueOnce( true )
vi.mocked( fs.readJson ).mockResolvedValueOnce({
endpoints: [ '/api/app/spotify.json' ]
} )
const data = await fetchJsonWithRetries( 'https://api.doesitarm.com/sitemap-endpoints.json', {
attempts: 2,
delayMs: 0
await expect( loadSitemapEndpoints() ).resolves.toEqual({
endpoints: [ '/api/app/spotify.json' ]
})
expect( data ).toEqual({
ok: true
} )
expect( axiosGet ).toHaveBeenCalledTimes( 2 )
expect( getJson ).not.toHaveBeenCalled()
} )
it( 'does not retry non-5xx errors', async () => {
const axiosGet = vi.mocked( axios.get )
it( 'falls back to the remote sitemap-endpoints JSON when the local file is missing', async () => {
vi.mocked( fs.pathExists ).mockResolvedValueOnce( false )
vi.mocked( getJson ).mockResolvedValueOnce({
endpoints: [ '/api/app/electron-framework.json' ]
} )
process.env.PUBLIC_API_DOMAIN = 'https://api.doesitarm.com'
axiosGet.mockRejectedValueOnce({
response: {
status: 404
}
await expect( loadSitemapEndpoints() ).resolves.toEqual({
endpoints: [ '/api/app/electron-framework.json' ]
} )
await expect( fetchJsonWithRetries( 'https://api.doesitarm.com/sitemap-endpoints.json', {
expect( getJson ).toHaveBeenCalledWith( 'https://api.doesitarm.com/sitemap-endpoints.json', {
attempts: 3,
delayMs: 0
} ) ).rejects.toEqual({
response: {
status: 404
}
})
expect( axiosGet ).toHaveBeenCalledTimes( 1 )
} )
it( 'classifies retryable server errors', () => {
expect( shouldRetryError( {
response: {
status: 502
}
} ) ).toBe( true )
expect( shouldRetryError( {
response: {
status: 503
}
} ) ).toBe( true )
expect( shouldRetryError( {
response: {
status: 404
}
} ) ).toBe( false )
expect( shouldRetryError( new Error( 'network' ) ) ).toBe( false )
delayMs: 1000
})
} )
} )

29
test/scanner/plist.test.ts
View file

@ -1,20 +1,16 @@
import { Buffer } from 'buffer'
import {
describe,
expect,
it,
vi
} from 'vitest'
import { describe, expect, it, vi } from 'vitest'
import {
parseFileSync,
parsePlistBuffer
parsePlistBuffer,
} from '~/helpers/scanner/parsers/plist-parser'
type ParsedPlist = Record<string, string>
const xmlPlist = Buffer.from( [
const xmlPlist = Buffer.from(
[
'<?xml version="1.0" encoding="UTF-8"?>',
'<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">',
'<plist version="1.0">',
@ -24,13 +20,18 @@ const xmlPlist = Buffer.from( [
' <key>CFBundleIdentifier</key>',
' <string>com.doesitarm.playwright-native-app</string>',
'</dict>',
'</plist>'
].join( '\n' ), 'utf8' )
'</plist>',
].join('\n'),
'utf8',
)
describe('plist parser', () => {
it('parses xml plist buffers asynchronously', async () => {
const callback = vi.fn()
const plist = await parsePlistBuffer( xmlPlist as any, callback ) as ParsedPlist
const plist = (await parsePlistBuffer(
xmlPlist as any,
callback,
)) as ParsedPlist
expect(plist.CFBundleExecutable).toBe('Playwright Native App')
expect(plist.CFBundleIdentifier).toBe('com.doesitarm.playwright-native-app')
@ -45,8 +46,8 @@ describe( 'plist parser', () => {
})
it('rejects invalid plist data', async () => {
await expect( parsePlistBuffer( Buffer.from( 'not-a-plist', 'utf8' ) as any ) )
.rejects
.toThrow( /Invalid binary plist/i )
await expect(
parsePlistBuffer(Buffer.from('not-a-plist', 'utf8') as any),
).rejects.toThrow(/Invalid binary plist/i)
})
})