Since this would be a change to core, do you mind creating an RFC? I agree that this would be a useful thing to be able to test for, and it would be especially nice for cases where one might want to use custom keyword validators (which we don't support now, but could be useful to support in the future!).

Thoughts on figuring out a way to include these in the invalid cases array? I'm not sure there's much value in adding a new error array, and I believe this could be done in a backwards-compatible fashion. As a quick illustration of what I'm envisioning, I think something like the following might allow us to use the current API (though changes would need to happen under the hood):

var rule = require("../../../lib/rules/my-rule");
var RuleTester = require("eslint").RuleTester;

var ruleTester = new RuleTester();
ruleTester.run("my-rule", rule, {
    valid: ["bar", "baz"],
    invalid: [
        {
            code: "foo",
            options: ["some-option", { someOtherOption: true }]
            errors: [
                {
                    messageId: "invalidConfiguration":
                    data: {
                        message: "Expected configuration option someOtherOption to be a string. Found boolean instead."
                    }
                }
            ]
        }
    ]
});

@kaicataldo sure, I will work on putting an RFC together.

Using the invalid array sounds good. In the interest of keeping this lightweight and leaving out irrelevant properties, what do you think about:

1. Disallowing code from being passed when the config is being tested (since it can't affect the results).
2. Encouraging messageId: "invalidConfiguration" to be used by itself without the exact message string. I'm not trying to test JSON Schema and don't really care what message it outputs. In fact, we might even want to disallow testing the exact string so that future JSON Schema string changes won't break our test cases.

{
    invalid: [
        {
            options: ["some-option", { someOtherOption: true }],
            errors: [
                {
                    messageId: "invalidConfiguration"
                }
            ]
        }
    ]
}

Both your points make a lot of sense to me!

Unfortunately, it looks like there wasn't enough interest from the team
or community to implement this change. While we wish we'd be able to
accommodate everyone's requests, we do need to prioritize. We've found
that issues failing to reach accepted status after 21 days tend to
never be accepted, and as such, we close those issues.
This doesn't mean the idea isn't interesting or useful, just that it's
not something the team can commit to.

Thanks for contributing to ESLint and we appreciate your understanding.

Reopening this as something I may try to write an RFC for at some point.

I have opened an RFC for this:

• feat: support for testing invalid rule schemas and runtime exceptions rfcs#103

moved to "ready for implement", as the rfc has been merged. :)