@rory I find it helps to keep a track recording in the background. Otherwise the location it records can be quite out of date. I think JOSM has a sync tool if your GPS is separate.
Also you don't HAVE to upload to Mapillary when you use their app.
What the other user says: log your track continuously and take your photos (or capture video) as you go, #geotagged or not. A number of post-processing applications allow to correlate the photos to the track using the timestamps.
I haven't done this recently myself but can give more details if needed.