Creating a paginated fetcher with React Hooks

In this post I will be detailing how I tackled creating a custom hook in React that abstracts the complexities of dealing with fetching paginated data for an infinite-scroll experience.

The user experience

You'll likely be familiar with the typical infinite scroll list-view user-experience that's commonplace on mobile, however I'll list out the requirements for the eventual end-user experience for brevity:

  • Initial loading spinner state when first visiting the interface and there are no results yet.
  • Lazy-load / infinite scroll to fetch more results when nearing the end of the list.
  • Loading state for when more results are loading.
  • Pull-to-refresh when at the top of the list and scrolling up.

The problem

Despite the user-experience being pretty simple, dealing will all of the states for fetching paginated data is actually pretty difficult. Let's take a look at a few of the states and methods needed to express the user-experience:

  • Loading state for when nothing has been fetched yet.
  • Loading state for when there are already results and the next page is loading.
  • Refreshing state for when the user issues a pull-to-refresh.
  • Error state for if a fetch goes wrong.
  • List of results that have been returned from fetching. For infinite scroll, the results should be appended to the existing list of results.
  • The total count of all records available from the back-end.
  • State for whether the user has reached the end of the list.
  • Method for fetching the initial page.
  • Method for refreshing the list.
  • Method for fetching the next page.

In an application that has many instances of paginated data, it makes sense to have a shared utility to manage and keep track of the state and provide these methods for managing paginated data. However, the utility can't know much about the actual data itself, or it would be difficult to be shared across many types of data. So, on top of the states above, the utility needs to have the following traits:

  • Can't know about the shape of the returned data. This means the utility can be flexible for different paginated data shapes.
  • Can't know where the data is stored once the data has been returned from the request. This means that the data can be cached.
  • Can't make any assumptions about the request to fetch data. This means any additional logic tied to fetching the data can be added outside of the utility, for example search, filtering and sorting that are managed outside of the pagination utility.

As you can see there's a lot of internal complexity for this kind of utility but the external API is flexible, simple and reusable. Let's get in to the code.

The code

I'll be building the paginated utility with React Hooks, and I'll assume that you're familiar with them in the rest of this post. If you're not, I highly recommend an afternoon with coffee and the official React documentation for hooks which do a much better job of explaining hooks than I can!

The external API

I typically start with the external API of an abstraction, which helps formulate a blueprint of what the internals requirements of it. Let's sketch something out:

const {
  ids,
  loadingInitial,
  loadingNextPage,
  refreshing,
  loaded,
  errored,
  endReached,
  fetchInitial,
  fetchNextPage,
  refresh
} = usePaginatedFetcher(
  (from, size) => () => fetchSomePaginatedData(from, size);
)

So I've sketched out the initial list of things that an external component using the paginated hook will need to use to implement the user experience described above. Here's the Typescript definitions:

{
  // Keeps a track of the ids returned from the fetches
  ids: string[];
  // Whether the data is loading for the first time
  loadingInitial: boolean;
  // Whether there's already some data and the next page is loading
  loadingNextPage: boolean;
  // Whether the user has issues a pull-to-refresh
  refreshing: boolean;
  // Whether any data has fetched, ever
  loaded: boolean;
  // Whether a fetch has errored
  errored: boolean;
  // Whether there are no more pages to fetch
  endReached: boolean;
  // Fetches the initial page, updates the loadingInitial state
  fetchInitial: () => Promise<...>;
  // Fetches the next page, updates the loadingNextPage state
  fetchNextPage: () => Promise<...>;
  // Refreshes the data, updates the refreshing state
  refresh: () => Promise<...>;
}

The reason that fetchInitial, fetchNextPage and refresh do not have any arguments is because these functions are inferred from the ReturnType of the passed fetcher function. More on the specifics of how this is achieved later in the post.

The last thing to map out is what the fetcher function looks like. The usePaginatedFetcher hook takes in a function as the first argument that is responsible for actually fetching the data from somewhere. This function receives from and size as the first two arguments (which represent which index of records the back-end should start returning, and how many records to return respectively). The fetcher is then expected to return another function that performs the fetching.

The reason the fetcher is structured as a higher-order function is because the hook is responsible for managing the state of from and size, but there could be any number additional arguments required to perform the fetching of the data, like ids, filters, searches, sorts, etc... and the hook shouldn't make any assumptions or restrict the flexibility of the data fetching itself. Here's the type definition of the fetcher function:

type Fetcher = Fn extends (from: number, size: number) => (...args: any[]) => Promise<Paginated<any>>

Where Paginated is an interface defined as:

interface Paginated<T> {
  count: number;
  results: T[];
}

Contradictory to what I stated above where the paginated fetcher hook mustn't know about the shape of the returned data, I am assuming that the API returns a response in the Paginated structure. Of course, your app will likely be different to this so feel free to amend as necessary, or refactor the hook to be even more generic.

So now that the external API is defined, let's move on to the internals.

Handling state

First things first, let's set up a bunch of useState hooks and derived state to manage the internal (and some exposed) states:

type Fetcher = Fn extends (from: number, size: number) => (...args: any[]) => Promise<Paginated<any>>

const usePaginatedFetcher = <Fn extends Fetcher>(fetch: Fn) => {
  const [ids, setIds] = useState<string[]>([]);
  const [count, setCount] = useState(0);
  const [loaded, setLoaded] = useState(false);
  const [errored, setErrored] = useState(false);
  const [loadingInitial, setLoadingInitial] = useState(false);
  const [refreshing, setRefreshing] = useState(false);
  const [loadingNextPage, setLoadingNextPage] = useState(false);
  const endReached = ids.length >= count;

  return {
    ids,
    loaded,
    errored,
    loadingInitial,
    refreshing,
    loadingNextPage,
    endReached
  }
}

Though we haven't tied any logic to these states yet, but by defining them, we've provided 7 out of 10 of our external APIs!

Fetching data

And fetching data goes as follows:

const fetchData = useCallback(
  (initial: boolean) => async (...args: any[]) => {
    const from = initial ? 0 : ids.length;
    try {
      const response = await fetch(from, size)(...args);
      const newIds = response.results.map(result => result[idKeyName]);
      setIds(initial || newIds.length === 0 ? newIds : [...ids, ...newIds]);
      setCount(response.count);
      setLoaded(true);
      return response;
    } catch (err) {
      setErrored(true);
    }
  },
  [fetch, ids.length, idKeyName]
);

There's a couple things to note here. The first is that the fetchData function is a higher-order function that takes as it's first argument initial: boolean. This boolean is required because the hook exposes methods fetchInitial and refresh, both of which which when called fetches data and replaces the existing list, aka, doesn't infinite scroll. The initial boolean is used within the setIds call to determine whether to append the results to the existing list (in the case of infinite scroll) or replace the results in the list.

The second thing to note is that there's a variable I haven't introduced yet, idKeyName. The hook isn't allowed to know about the structure of the returned data, yet it's expected to keep track of the list of results to ve displayed in the UI. These two things are contradictory of course, so there needs to be some way for the hook to know how to uniquely identify a particular result. To achieve this, a idKeyName is passed in as a new argument to the hook and is used to map a unique identifier per record. The list of these identifiers are managed in the state of the hook and passed back out so that the component can map over them and render the results:

const usePaginatedFetcher = <Fn extends Fetcher>(fetch: Fn, idKeyName: string) => ...

Fetching the next page, refreshing etc

Now a reusable fetchData higher-order function has been defined, it's simple to create some fetchers tied to individual states and logic:

const fetchInitial = useCallback(
  async (...args: any[]) => {
    if (!loadingInitial) {
      setLoaded(false);
      setLoadingInitial(true);
      try {
        return await fetchData(true)(...args);
      } finally {
        setLoadingInitial(false);
      }
    }
  },
  [fetchData, loadingInitial]
) as Return;

const refresh = useCallback(
  async (...args: any[]) => {
    if (!refreshing) {
      setRefreshing(true);
      try {
        return await fetchData(true)(...args);
      } finally {
        setRefreshing(false);
      }
    }
  },
  [fetchData, refreshing]
) as Return;

const fetchNextPage = useCallback(
  async (...args: any[]) => {
    if (!loadingNextPage && !endReached) {
      setLoadingNextPage(true);
      try {
        return await fetchData(false)(...args);
      } finally {
        setLoadingNextPage(false);
      }
    }
  },
  [fetchData, loadingNextPage, endReached]
) as Return;

Where did as Return come from? Well since these methods are called externally, it's important that they represent the shape of the fetcher function omitting the higher-order bit (where the from and size arguments are injected) that the hook calls with the state of the from and size properties. In fact, Return should be the return type of the fetcher function argument explained above. Let's drill it down.

The fetcher function type looks like this:

type Fetcher = Fn extends (from: number, size: number ) => (...args: any[]) => Promise<Paginated<any>>

However, we want the methods exposed by the hook to look something like this:

type FetchNextPage = (...args: any[]) => Promise<Paginated<any>>;

With the arguments inferred of course, such that if the fetcher function requires any additional arguments such as ids, search, filter sort etc, the callee of the fetcher knows about it. Typescript comes with a handy type utility called ReturnType that we can use to our advantage:

export const usePaginatedFetcher = <Fn extends Fetcher>(
  fetch: Fn,
  idKeyName: string
) => {
  type Return = ReturnType<typeof fetch>;
  ...
}

Pretty neat!

Stitching it all together

Hopefully you'll get a sense for how all the bits of this hook fit together, but here's the entire code for the custom paginated data hook!

type Fetcher = (
  from: number,
  size: number
) => (...args: any[]) => Promise<Paginated<any>>;

export const usePaginatedFetcher = <Fn extends Fetcher>(
  fetch: Fn,
  idKeyName: string,
  size = 30
) => {
  type Return = ReturnType<typeof fetch>;

  const [ids, setIds] = useState<string[]>([]);
  const [count, setCount] = useState(0);
  const [loaded, setLoaded] = useState(false);
  const [errored, setErrored] = useState(false);
  const [loadingInitial, setLoadingInitial] = useState(false);
  const [refreshing, setRefreshing] = useState(false);
  const [loadingNextPage, setLoadingNextPage] = useState(false);

  const endReached = ids.length >= count;

  const fetchData = useCallback(
    (initial: boolean) => async (...args: any[]) => {
      const from = initial ? 0 : ids.length;
      try {
        const response = await fetch(from, size)(...args);
        const newIds = response.results.map(result => result[idKeyName]);
        setIds(initial || newIds.length === 0 ? newIds : [...ids, ...newIds]);
        setCount(response.count);
        setLoaded(true);
        return response;
      } catch (err) {
        setErrored(true);
      }
    },
    [fetch, ids.length, idKeyName]
  );

  const fetchInitial = useCallback(
    async (...args: any[]) => {
      if (!loadingInitial) {
        setLoaded(false);
        setLoadingInitial(true);
        try {
          return await fetchData(true)(...args);
        } finally {
          setLoadingInitial(false);
        }
      }
    },
    [fetchData, loadingInitial]
  ) as Return;

  const refresh = useCallback(
    async (...args: any[]) => {
      if (!refreshing) {
        setRefreshing(true);
        try {
          return await fetchData(true)(...args);
        } finally {
          setRefreshing(false);
        }
      }
    },
    [fetchData, refreshing]
  ) as Return;

  const fetchNextPage = useCallback(
    async (...args: any[]) => {
      if (!loadingNextPage && !endReached) {
        setLoadingNextPage(true);
        try {
          return await fetchData(false)(...args);
        } finally {
          setLoadingNextPage(false);
        }
      }
    },
    [fetchData, loadingNextPage, endReached]
  ) as Return;

  return {
    ids,
    loadingInitial,
    loadingNextPage,
    refreshing,
    loaded,
    errored,
    endReached,
    fetchInitial,
    fetchNextPage,
    refresh
  };
};

Next steps

Of course, this paginated data hook isn't perfect. For one, it relies on the response of the data from the back-end being consistent, and makes some assumptions based on it, and it cannot be pre-loaded with a list of ids, so the first page will always be fetched, even if there's already some cached state from previous fetches lying around in a global store. However hopefully this gives you an idea of how React Hooks can be used to create powerful and useful abstractions for what's normally a pretty difficult state management problem to share across many components easily.