Preheat your oven to 450°F. Spray a 9 inch round glass pie plate with olive oil or cooking spray. Rub with the cut sides of the garlic. Discard the garlic.
In a pan, combine 2 oz. of the goat cheese, the milk and the thyme, and bring just to a simmer. Remove from heat and remove the thyme stems.
While the milk is heating, layer half of the potato slices in the pie plate, drizzle with 1/2 Tablespoon olive oil, 1/4 teaspoon salt, and black pepper. Repeat with one layer of sweet potatoes, then another layer of white potatoes.
Pour the milk mixture over the top of the potatoes, and lay the thyme sprigs on top.
Cover with aluminum foil and bake for 20 minutes.
Uncover, crumble the remaining 2 oz. of goat cheese and sprinkle on the top.
Bake uncovered for another 20-25 minutes, or until the potatoes are tender and the cheese starts to brown.
You can use all white or all sweet potatoes or a different amount of each for a total of about 1 1/2-1 3/4 lbs total.