Skip to content

Conversation

@kevkevinpal
Copy link
Contributor

@kevkevinpal kevkevinpal commented Aug 9, 2025

Description

related and closes #2

I wanted a way to retrieve historical estimates.
This PR introduces a new endpoint GET /historical_fee that takes the date query parameter to query for historical fee estimates.

Details

Add date=<utc timestamp> query param

GET 
/historical_fee?date=<utc timestamp>

will get you the same response as GET /fees but with an estimation from the provided date

Example request

http://100.97.107.124:8080/historical_fee?date=1755353169

Example response
{
  "mempool_update_time" : "2025-08-08T15:10:10.639Z",
  "estimates" : {
    "3" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 2.0536
        },
        "0.20" : {
          "fee_rate" : 3.8163
        },
        "0.50" : {
          "fee_rate" : 4.2922
        },
        "0.80" : {
          "fee_rate" : 5.0075
        },
        "0.95" : {
          "fee_rate" : 6.0853
        }
      }
    },
    "6" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.9226
        },
        "0.20" : {
          "fee_rate" : 2.3777
        },
        "0.50" : {
          "fee_rate" : 3.8457
        },
        "0.80" : {
          "fee_rate" : 4.8669
        },
        "0.95" : {
          "fee_rate" : 5.9614
        }
      }
    },
    "9" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.5556
        },
        "0.20" : {
          "fee_rate" : 1.9986
        },
        "0.50" : {
          "fee_rate" : 3.0743
        },
        "0.80" : {
          "fee_rate" : 4.7505
        },
        "0.95" : {
          "fee_rate" : 5.1867
        }
      }
    },
    "12" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0186
        },
        "0.20" : {
          "fee_rate" : 1.8217
        },
        "0.50" : {
          "fee_rate" : 2.8466
        },
        "0.80" : {
          "fee_rate" : 3.692
        },
        "0.95" : {
          "fee_rate" : 4.6578
        }
      }
    },
    "18" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0154
        },
        "0.20" : {
          "fee_rate" : 1.7262
        },
        "0.50" : {
          "fee_rate" : 2.4272
        },
        "0.80" : {
          "fee_rate" : 3.4497
        },
        "0.95" : {
          "fee_rate" : 4.2684
        }
      }
    },
    "24" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.014
        },
        "0.20" : {
          "fee_rate" : 1.6423
        },
        "0.50" : {
          "fee_rate" : 1.7875
        },
        "0.80" : {
          "fee_rate" : 3.2517
        },
        "0.95" : {
          "fee_rate" : 3.4585
        }
      }
    },
    "36" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0
        },
        "0.20" : {
          "fee_rate" : 1.4825
        },
        "0.50" : {
          "fee_rate" : 1.521
        },
        "0.80" : {
          "fee_rate" : 2.0404
        },
        "0.95" : {
          "fee_rate" : 2.9985
        }
      }
    },
    "48" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0
        },
        "0.20" : {
          "fee_rate" : 1.3649
        },
        "0.50" : {
          "fee_rate" : 1.3848
        },
        "0.80" : {
          "fee_rate" : 1.741
        },
        "0.95" : {
          "fee_rate" : 2.7334
        }
      }
    },
    "72" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0
        },
        "0.20" : {
          "fee_rate" : 1.1532
        },
        "0.50" : {
          "fee_rate" : 1.2032
        },
        "0.80" : {
          "fee_rate" : 1.2808
        },
        "0.95" : {
          "fee_rate" : 1.5801
        }
      }
    },
    "96" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0
        },
        "0.20" : {
          "fee_rate" : 1.0385
        },
        "0.50" : {
          "fee_rate" : 1.0821
        },
        "0.80" : {
          "fee_rate" : 1.1039
        },
        "0.95" : {
          "fee_rate" : 1.2255
        }
      }
    },
    "144" : {
      "probabilities" : {
        "0.05" : {
          "fee_rate" : 1.0
        },
        "0.20" : {
          "fee_rate" : 1.0
        },
        "0.50" : {
          "fee_rate" : 1.0
        },
        "0.80" : {
          "fee_rate" : 1.0101
        },
        "0.95" : {
          "fee_rate" : 1.0202
        }
      }
    }
  }
}

TODO

Note this is still a draft, I still need to do a few things, but wanted to put it out there for any draft staged review

  • write some testing
  • clean up the code

Follow-up:
Create another endpoint for multiple historical fees (rebase after this merges) #11
GET /historical_fees?start_date=<utc timestamp>&end_date=<utc timestamp>&interval=<1h,20m,30s>

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from 55ae962 to c176e20 Compare August 9, 2025 16:49
@kevkevinpal
Copy link
Contributor Author

Btw, I'm using this to create a chart like this, and my goal is to then compare the estimates to what happened on-chain to be able to chart out weekly the accuracy with live data.

I also just think it would be useful to be able to reference historical estimations

One final thought is that Augur currently does not support sub 1 sat/vbyte transactions and Bitcoin-Core should be dropping its mintxrelay amount bitcoin/bitcoin#33106, so it might be useful to also do the same for Augur

Screenshot 2025-08-11 at 10 04 34 AM

@rloomba
Copy link
Collaborator

rloomba commented Aug 11, 2025

nice work @kevkevinpal! this is definitely a change we'll want to merge in.

some feedback:

  • what do you think about creating a new endpoint? maybe historical_fees so we can experiment and ideate on this new use case, while leaving fees untouched
  • any reason you went with just a timestamp vs a range laid out in [Idea] Expose mempool historical data via and endpoint #2? the timestamp is simpler and works, but just wondering

@laurenshareshian
Copy link
Collaborator

Just a heads up that I'm going to take this out of draft mode so that I can follow the conversation between you and @rloomba. We have some GitHub slack notifications in place that I'm pretty sure don't notify us when things are in draft mode, and we'd like to be able to follow new activity 😄

@laurenshareshian laurenshareshian marked this pull request as ready for review August 11, 2025 22:53
@kevkevinpal
Copy link
Contributor Author

  • what do you think about creating a new endpoint? maybe historical_fees so we can experiment and ideate on this new use case, while leaving fees untouched

1). Yea I can do that, was not sure if a new endpoint or building on the current one was preferred. Either is fine for me


2). I think when implementing the date param was just simpler and fit how GET /fees already worked, but the range might be better. Especially if a user needs to fetch multiple historical dates at the same time.

Maybe having multiple two endpoints might make sense
GET /historical_fee?date=YYYY-MM-DDTHH:MM:SS
and a plural version
GET /historical_fees?start_date=YYYY-MM-DDTHH:MM:SS&end_date=YYYY-MM-DDTHH:MM:SS&interval=<1h,20m,30s>
With interval being optional and set at 1 hour by default?

I can work on the singular version for this PR and open another PR for a plural/multiple historical fees version

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch 3 times, most recently from cb70f55 to fa51beb Compare August 12, 2025 00:42
@kevkevinpal
Copy link
Contributor Author

So I went ahead and created
GET /historical_fee?date=YYYY-MM-DDTHH:MM:SS in fa51beb
and
GET /historical_fees?start_date=YYYY-MM-DDTHH:MM:SS&end_date=YYYY-MM-DDTHH:MM:SS&interval=<int in seconds> in 977c25c

let me know what you think and if there are any modifications you'd like me to do and I can work on it!

@kevkevinpal
Copy link
Contributor Author

Example response and use for GET /historical_fees

Screenshot 2025-08-11 at 9 54 49 PM

Screenshot 2025-08-11 at 9 56 09 PM

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from 977c25c to 3c3757d Compare August 12, 2025 11:59
@rloomba
Copy link
Collaborator

rloomba commented Aug 13, 2025

nice work on this @kevkevinpal! will review this in a day or two

@kevkevinpal kevkevinpal changed the title feat: added parament date to get estimate by specific UTC timestamp feat: GET /historical_fee and GET historical_fees Aug 13, 2025
*/
fun getFeeEstimateForDate(dateTimeUTC: LocalDateTime): FeeEstimate? {
val dateTime = dateTimeUTC
.atZone(ZoneId.of("UTC")) // Interpret as UTC
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the logic to handle timezones is overly complex, what do you think about just using UTC timestamps instead?

// Single historical estimate
GET /historical_fee?timestamp=1723128639

// Date range with interval in seconds
GET /historical_fees?start_timestamp=1723128639&end_timestamp=1723214639&interval=3600

@rloomba
Copy link
Collaborator

rloomba commented Aug 15, 2025

fcc12bc looks pretty good to me. What do you think about breaking that into it's own PR so we can get that merged quicker. I think it just needs:

for /historical_fees, I think we'll need to think about pagination (what happens if we want large date ranges?) and do some further performance testing

@kevkevinpal
Copy link
Contributor Author

I pushed this commit to use timestamps instead fae5612

I also dropped the last commit and opened this PR which I can rebase onto main after this merges #11

Will try to write up a test aswell

@kevkevinpal kevkevinpal changed the title feat: GET /historical_fee and GET historical_fees feat: GET /historical_fee Aug 16, 2025
@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from fae5612 to eac411d Compare August 16, 2025 19:49
@kevkevinpal
Copy link
Contributor Author

I pushed tests in eac411d

let me know if you want me to squash e66b5b3 with eac411d to have the test and implementation in the same commit

Also, let me know if you want any additional tests added!

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from eac411d to e765828 Compare August 16, 2025 22:46
@rloomba
Copy link
Collaborator

rloomba commented Aug 16, 2025

This looks great, only feedback is, let's change all variable names from date to timestsamp so we're super clear.

I'm thinking:

date -> timestamp
dateTimeUTC -> timestampUTC
getFeeEstimateForDate -> getFeeEstimateForTimestamp
parse date -> parse timestamp

@rloomba
Copy link
Collaborator

rloomba commented Aug 16, 2025

and let's get some example curl commands for this endpoint added to the README

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from e765828 to a65f8ff Compare August 16, 2025 23:00
@kevkevinpal
Copy link
Contributor Author

awesome ya that makes sense to me as well, I updated to using timestamp in b6f2f50 and the tests in a65f8ff

@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from a65f8ff to ef26ad2 Compare August 16, 2025 23:02
@kevkevinpal kevkevinpal requested a review from rloomba August 16, 2025 23:06
Copy link
Collaborator

@rloomba rloomba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically ready for merge! one final task:

  • example curl request/response in the README.MD

Signed-off-by: kevkevinpal <oapallikunnel@gmail.com>
Signed-off-by: kevkevinpal <oapallikunnel@gmail.com>
Signed-off-by: kevkevinpal <oapallikunnel@gmail.com>
@kevkevinpal kevkevinpal force-pushed the historicalEstimateFees branch from 59782d5 to 473787b Compare August 16, 2025 23:53
@kevkevinpal
Copy link
Contributor Author

This is basically ready for merge! one final task:

  • example curl request/response in the README.MD

oops ya just added an example request/response here

473787b

@kevkevinpal kevkevinpal requested a review from rloomba August 17, 2025 00:06
Copy link
Collaborator

@rloomba rloomba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice job!!

@rloomba rloomba merged commit 6384f11 into block:main Aug 17, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Idea] Expose mempool historical data via and endpoint

3 participants