Apache 2.4 Require ip not working

Trying to go from older allow, deny, order syntax to the new one to secure WordPress admin section, but I can't get it to recognize my IP.

This is what my .htaccess file contains in /wp-admin folder.

ErrorDocument 401 default
ErrorDocument 403 default

# Disallow access for everyone except these IPs
<RequireAny>
    Require ip 50.153.218.4
</RequireAny>

# Allow plugin access to admin-ajax.php around password protection
<Files admin-ajax.php>
    <RequireAll>
        Require all granted
    </RequireAll>
</Files>

And this is what I have in .htaccess in the root under the WordPress mod rewrite info.

# Protect WordPress
ErrorDocument 401 default
ErrorDocument 403 default

<Files wp-login.php>
    <RequireAny>
        Require ip 50.153.218.4
    </RequireAny>
</Files>

But I just keep getting 403 Forbidden error. When I add Require All Granted under the IP, it works fine, but that opens it up to every user. It seems like apache is just not reading my ip correctly? Anyone have any idea what I'm doing wrong?