Instead of maintaining two HashMaps, one for window and the other one for expected frequency, we can use just one.

Initialize the map with the pattern. Every time when a character enters a window, remove it from the pattern map. Every time a character leaves the window, add it to the map.

When count for any character becomes 0, we simply remove the character from the map.

This way we are done when the map becomes empty.

Eventually the number of map lookups becomes much smaller. Even hash map lookup is expected to be O(1), it still is relatively expensive.